evilrouters.net

Several months ago, I wrote about discovering an “anomaly” in ProCurve firmware version K.13.45. Because I apparently already have some folks at HP pissed off, I thought I’d take a look and see if anything had changed.

Although the case hasn’t been updated since January 5th and still hasn’t been resolved, it is still listed as “in progress”.

Since simply copy/pasting the output doesn’t really show the true nature of the “anomaly”, I thought I’d go ahead and put together a video. What I didn’t mention originally (and, perhaps, should have) is that CPU utilization skyrockets to 99% while these “show” commands are running (running very slowly, that is). Originally, and in this video, I used “show ip igmp config” to demonstrate, but there are other show commands that have the same effect on CPU.

In the video, I have two SSH sessions open to an HP ProCurve 5406zl switch running version K.13.60 firmware (yes, I know it’s not the latest — read this to find out why I’m not running K.13.63). In the top session, you see me execute these commands:

HP5406# sh system
HP5406# repeat delay 1

The first command runs “show system-information”. The second causes it to be repeatedly executed, in one second intervals. (I would’ve preferred “sh system | in CPU” since HP FINALLY gave us that feature, but apparently the piping isn’t taken into account when the “repeat” command is used.) I let this run for maybe ten seconds, to give a good “baseline” of what the CPU utilization of the switch is. It hovers mostly around 3-4%.

In the bottom session, you see me execute:

HP5406# show ip igmp config

Immediately after executing this show command, notice how SLOOOOOOW the output to the terminals become. It takes two or three iterations of the “sh system” command, but the switch quickly hits 99% CPU utilization. This, of course, is why the output is being taking so long to display. I’m not sure what’s happening behind the scenes, but apparently the ProCurve doesn’t like it. I cut the video off just a bit too quickly, but it should be noted that CPU utilization returns to normal immediately after the “show” command has finished executing.

Anyway, here’s the video:

I am unfortunate enough to have to maintain 30 or so HP ProCurve switches (with another 10 or so almost ready to be installed). I hate them, with a passion. It is impossible for me to accurately convey how much I hate them using mere words alone.

People often wonder why.

The reasons vary. I’ve had “support” respond to me that they cannot reproduce issues in the lab because they don’t have enough GBICs available (well, go fucking get them!). Their “solution” in this case was to offer me a fresh-off-the-compiler “beta” version of the switch firmware. Because I deal with production equipment (you know, stuff I want to work), I always pass. I am not HP’s QA department, nor will I act as such. Here’s why.

Today I was reading through the K.13.63 Release Notes and noticed this “known issue”:

“Transferring a switch configuration of 4,201 bytes or larger to a switch’s /cfg/startup-config directory via SCP will result in the switch coming up on factory defaults or with the new configuration only partly installed after reboot”

Are you fuckin’ kidding me, HP? How does shit like this make it past QA?

Looking at one of my 5400s, I see its startup config is 18,370 bytes. I have a lot of stuff to add to the config on that one, so it will likely easily pass 30k before it’s “done”. Glad I didn’t “upgrade” to this firmware earlier — I would’ve hated to have been the guy to find that issue.

There’s more that I’d share, but I’ll let you browse through the “Fixes” and “Known Issues” sections of the Release Notes for yourself. Look out for phrases like “hang”, “may reboot unexpectedly”, and the like.

Tomorrow, I’ll be receiving one of the new HP ProCurve Threat Management Services zl Modules. My HP sales rep has warned me in advance that “this module is not very intuitive, like many things you may have seen before”. Why am I not surprised?

From HP’s Advanced Traffic Management Guide:

“BPDU protection is a security feature designed to protect the active STP topology by preventing spoofed BPDU packets from entering the STP domain. In a typical implementation, BPDU protection would be applied to edge ports connected to end user devices that do not run STP. If STP BPDU packets are received on a protected port, the feature will disable that port and alert the network manager …”

In short, we want to enable BPDU protection on any “edge ports”, or ports that are connected to end user devices (PCs, printers, etc.) — any device that should not be sending out STP BPDU’s. Without BPDU protection, a malicious (or ignorant) user could plug a switch into our network and alter the spanning tree topology. That’s a bad thing.

With BPDU protection, we can a port automatically disabled if it receives an STP BPDU. Under normal circumstances, this should never happen. Enabling this feature does two things:

• helps protect your network, and
• keeps you, the network guy (or gal), informed about what is happening.

Let’s assume a user brings in their own switch from home. They unplug their PC from the wall, plug into their own switch, then plug their switch into the data jack in the wall. As soon as their switch sends out a BPDU, our switch will receive it and immediately disable the port. If you do not configure your switch to automatically re-enable the port after a specified period of time (I don’t), their data jack is effectively dead. It cannot be used again without the intervention of the network guy. When they can no longer work and must come to you for assistance, you’ll have a good idea of what they did and can promptly break their fingers educate them on why they should never do that.

On HP ProCurve switches, we enable BPDU protection with the “spanning-tree <port> bpdu-protection” command. You can do this individually per port (you can specify ranges) or do it like I do:

SWITCH(config)# spanning-tree all bpdu-protection

This, in my opinion, is the best way. This will enable BPDU protection on EVERY port. Then, you selectively disable it on your uplink ports:

SWITCH(config)# no spanning-tree a24 bpdu-protection

The default BPDU protection timeout is set to 0. This is the amount of time, in seconds, after which the switch will re-enable a port that has been disabled due to receiving an “illegal” BPDU. A value of 0 means “never”, and is the value that I prefer. This ensures that “network guy intervention” is required to break the user’s fingers re-enable the port.

Let’s take a look at what happens. In this example, I have BPDU protection enabled on port A2 (all ports except for A24, actually), as can be verified here:

SWITCH# sh spanning-tree bpdu-protection

 Status and Counters - STP Port(s) BPDU Protection Information

 BPDU Protection Timeout (sec) : 0                     
 BPDU Protected Ports : A1-A23,B1-B24,C1-C24,D1-D24,E1-E24,F1-F24,G1-G24,H1-H2...

So, having verified that BPDU protection is enabled on port A2, what happens when we plug another switch into that port? You obviously can’t see me plug in the cable, but here’s what we get in our logs:

SWITCH# sh logging
 Keys:   W=Warning   I=Information
         M=Major     D=Debug E=Error
----  Event Log listing: Events Since Boot  ----
I 03/12/09 00:02:41 00435 ports: port A2 is Blocked by STP
I 03/12/09 00:02:41 00840 stp: port A2 disabled - BPDU received on protected
            port.
I 03/12/09 00:02:41 00898 ports: BPDU protect(5) has disabled port A2 for 0
            seconds
I 03/12/09 00:02:41 00077 ports: port A2 is now off-line
----  Bottom of Log : Events Listed = 99  ----

Our switch will, when the port comes “up”, wait briefly and just listen. It will wait to see if it receives a BPDU (“Blocked by STP”) and, as soon as it does, notes this in the system log and immediately disables the port.

SWITCH# sh spanning-tree bpdu-protection a2

 Status and Counters - STP BPDU Protection Information

 BPDU Protection Timeout (sec) : 0                     
 BPDU Protected Ports : A1-A23,B1-B24,C1-C24,D1-D24,E1-E24,F1-F24,G1-G24,H1-H2..
.

  Port  Type      Protection State      Errant BPDUs
  ----- --------- ---------- ---------- ------------
  A2    100/1000T Yes        BpduError  1

The “sh spanning-tree bpdu-protection a2″ command shows us that port A2 is in “BpduError” state and has received a total of one “errant BPDUs”. Now, let’s assume we’ve unplugged the switch and plugged a PC back in. How do we get the port functional again? Easy:

SWITCH# conf
SWITCH(config)# interface a2
SWITCH(eth-A2)# enable
SWITCH(eth-A2)# end

That’s all there is to it! BPDU protection is simple to implement (assuming you know what connects where) and can add a bit of protection to your network. Turn it on!

Earlier this week, I described “Upgrading ProCurve firmware via TFTP”. TFTP has been the primary method of updating firmware on network devices for many, many years. TFTP, however, lacks authentication and encryption. HP has begun offering first FTP and now SFTP on some of their network devices as an alternative method for updating firmware. In this post, we will update the switch firmware using SFTP.

A few items to note:

• 192.168.1.113 is the IP address of the ProCurve 2610 (DHCP FTW!)
• BL-C234-AS01 is the hostname assigned to the switch (don’t ask)

Quick aside: I should point out that I don’t actually use DHCP on my switches. This is a brand new, straight out of the box switch I’m messing with. I’m doing this in the home lab, where DHCP is fine for this. Before the switch even gets close to being put into production at $work, it will be fully configured and have an IP address assigned for management. I got an e-mail asking why in hell I’d use DHCP, so I wanted to clear that up.

In the previous post, we upgraded the switch firmware in the primary flash on an HP ProCurve 2610 switch. We rebooted the switch and verified that it was now running the latest firmware:

BL-C234-AS01# show version
Image stamp:    /sw/code/build/nemo(ndx)
                Jan 14 2009 15:31:02
                R.11.25
                301
Boot Image:     Primary

We can quickly verify which versions of the firmware currently saved in flash:

BL-C234-AS01# show flash
Image           Size(Bytes)   Date   Version
-----           ----------  -------- -------
Primary Image   : 3790986   01/14/09 R.11.25 
Secondary Image : 3689315   01/25/08 R.11.07 
Boot Rom Version: R.10.06
Current Boot    : Primary

In this post, we are going to use SFTP to transfer the R.11.25 firmware from a Linux server to the secondary flash on the switch.

Quick aside: A much easier way to “sync” the primary and secondary flash is to use the command “copy flash flash <destination>”. “copy flash flash secondary” would copy the firmware in the primary flash to the secondary flash. I’m not going to use that method, however, because the whole purpose is to illustrate the use of SFTP.

The first thing we need to do is to create a “manager” user account. We will use this account when we connect to the switch using SFTP. Here, I’m using “admin” for both the username and password. I don’t do this in real life, and neither should you:

BL-C234-AS01# conf
BL-C234-AS01(config)# password manager user-name admin 
New password for Manager: *****
Please retype new password for Manager: *****

Now that we have a user account set up, we need to enable SSH, then SFTP for transferring files:

BL-C234-AS01(config)# ip ssh
BL-C234-AS01(config)# ip ssh filetransfer 
Tftp and auto-tftp have been disabled.

Note that the switch automatically disabled TFTP and auto-TFTP for us when enabling SFTP. At this point, we should be able to use SFTP to connect to the switch. The first line of output from “show ip ssh” confirms for us that both SSH and Secure Copy (HP’s term for it) are enabled:

BL-C234-AS01# show ip ssh

  SSH Enabled     : Yes                 Secure Copy Enabled : Yes
  TCP Port Number : 22                  Timeout (sec)       : 120  
  Host Key Type   : RSA                 Host Key Size       : 2048

  Ciphers : aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc,
            rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
  MACs    : hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96

  Ses Type     | Protocol  Source IP and Port   
  --- -------- + --------- ---------------------
  1   console  |
  2   inactive |
  3   inactive |
  4   inactive |

Let’s try to connect:

$ sftp admin@192.168.1.113
Connecting to 192.168.1.113...
The authenticity of host '192.168.1.113 (192.168.1.113)' can't be established.
RSA key fingerprint is af:a6:20:67:64:8a:64:bf:7a:72:0e:52:c1:27:a6:4c.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.113' (RSA) to the list of known hosts.
We'd like to keep you up to date about:
  * Software feature updates
  * New product announcements
  * Special events

Please register your products now at:  www.ProCurve.com

admin@192.168.1.113's password:
sftp>

Now that we’re in, let’s take a look at the filesystem structure:

sftp> ls -l
drwxr-xr-x    2 J9088A   J9088A          0 JAN 01 00:00 cfg
drwxr-xr-x    2 J9088A   J9088A          0 JAN 01 00:00 log
drwxr-x---    2 J9088A   J9088A          0 JAN 01 00:00 os
drwxr-x---    3 J9088A   J9088A          0 JAN 01 00:00 ssh
sftp>

The “cfg” directory holds the running-config and startup-config files. If you wanted to create your startup-config elsewhere (or had a template that you use for new devices), you could upload it there.

The “os” directory is the one we are concerned with here, however. Let’s look at what’s contained there:

sftp> cd os
sftp> ls -l
-rwxrw----    1 J9088A   J9088A    3790986 JAN 14  2009 primary
-rwxrw----    1 J9088A   J9088A    3689315 JAN 25  2008 secondary
sftp>

We see two files: “primary” and “secondary”. Note that the filesize corresponds to our earlier output from “show flash”:

BL-C234-AS01# show flash
Image           Size(Bytes)   Date   Version
-----           ----------  -------- -------
Primary Image   : 3790986   01/14/09 R.11.25 
Secondary Image : 3689315   01/25/08 R.11.07 
Boot Rom Version: R.10.06
Current Boot    : Primary

We want to upload our local file, R_11_25.swi, to a file named “secondary” on the switch (note that you cannot create new files on the switch’s internal filesystem). This is accomplished easily enough:

sftp> put R_11_25.swi secondary
Uploading R_11_25.swi to /os/secondary
R_11_25.swi                                                                           100% 3702KB 528.9KB/s   00:07
sftp> exit
$

Looks like it went okay. Trust, but verify:

BL-C234-AS01# show flash
Image           Size(Bytes)   Date   Version
-----           ----------  -------- -------
Primary Image   : 3790986   01/14/09 R.11.25 
Secondary Image : 3790986   01/14/09 R.11.25 
Boot Rom Version: R.10.06
Current Boot    : Primary

Success!

I had a new HP ProCurve 2610 arrive today, and one of the first things I do to new devices is load up the latest firmware. I thought I’d document the procedure since 1) I’m about to do just that, and 2) my post “Upgrading HP Procurve firmware via USB flash drive” gets quite a few hits.

A few items to note:

• 192.168.1.12 is the IP address of my TFTP server
• 192.168.1.113 is the IP address of the ProCurve 2610 (DHCP FTW!)
• BL-C234-AS01 is the hostname assigned to the switch (don’t ask)

This is a new device, so I don’t have any of the firmware for this product line already on my local TFTP server. The first thing I do is to make sure I have a local copy of that:

First, I’ll take a look at what version of firmware the switch shipped with (and already exists, therefore, on flash):

BL-C234-AS01# show version
Image stamp:    /sw/pre/build/nemo(ndx)
                Jan 25 2008 13:54:14
                R.11.07
                104
Boot Image:     Primary
BL-C234-AS01# show flash
Image           Size(Bytes)   Date   Version
-----           ----------  -------- -------
Primary Image   : 3689315   01/25/08 R.11.07 
Secondary Image : 3689315   01/25/08 R.11.07 
Boot Rom Version: R.10.06
Current Boot    : Primary

We see that’s we’re running firmware version R.11.07. When copying the firmware to my TFTP server, I’ll use the same filename convention that HP does, so our filename will be R_11_07.swi. Let’s save a copy to the TFTP server:

BL-C234-AS01# copy flash tftp 192.168.1.12 R_11_07.swi

I always like to verify that the file actually shows up on the TFTP server, regardless of any error messages (or lack thereof) from the switch:

$ ls -l /tftpboot/R_11_07.swi
-rw-r--r-- 1 nobody nogroup 3689315 2009-02-02 23:28 /tftpboot/R_11_07.swi

In the meantime, I’ve downloaded the latest version of the software, R.11.25, from HP’s FTP server and saved it as R_11_25.swi on the TFTP server. Let’s go ahead and copy it over to the primary flash on the switch:

BL-C234-AS01# copy tftp flash 192.168.1.12 R_11_25.swi primary
The Primary OS Image will be deleted, continue [y/n]?  y

Here we tell the switch to copy from a TFTP server to flash memory, the TFTP server has IP address 192.168.1.12, the filename on the TFTP server is R_11_25.swi, and that we want that file saved to the primary flash on the switch. You’ll see a progress counter as the file is transferred, then:

Validating and Writing System Software to FLASH...

which takes a moment. After it has completed, we can verify success by examining the contents of flash:

BL-C234-AS01# show flash
Image           Size(Bytes)   Date   Version
-----           ----------  -------- -------
Primary Image   : 3790986   01/14/09 R.11.25 
Secondary Image : 3689315   01/25/08 R.11.07 
Boot Rom Version: R.10.06
Current Boot    : Primary

We can now reboot the switch with the new firmware:

BL-C234-AS01# reload
Device will be rebooted, do you want to continue [y/n]?  y

After the switch boots back up, we can verify that it is running the latest firmware:

BL-C234-AS01# show version
Image stamp:    /sw/pre/build/nemo(ndx)
                Jan 14 2009 15:31:02
                R.11.25
                301
Boot Image:     Primary

HINT: Always read the release notes before upgrading firmware, especially on a production device!