Image of Cortney & Jeremy

Why Is Cisco Crawling My Website?

by Jeremy L. Gaddis on August 24, 2011 · 6 comments

in Website

A while back, I noticed my website was being hit by a new crawler at IP address 72.163.33.126:

$ host 72.163.33.126
126.33.163.72.in-addr.arpa domain name pointer srch-prod-170.cisco.com.

Cisco has apparently decided to keep an eye on me. Okay, whatever, that’s fine, but since you never know what Cisco’s up to, I decided to keep my eye on them too.

Then, something happened. On August 10th, the crawler showed up in the logs just as it normally did. At 16:43:39 (UTC, of course) it hit the home page and started making its way through the site, hitting a new page every 10 seconds. At 18:09:54 it makes its last request of the day and then calls it quits.

Unlike normal, though, it didn’t show up again the next day. Or the day after that. Or the day after that. Or …

What Happened to the Crawler?

When the crawler didn’t show up for work again I was left wondering what had happened. After glancing at my calendar for a moment, though, it hit me like a viagra!

On the afternoon of August 10th, Cisco held a conference call to announce their 4Q and FY11 earnings.

Shortly after the call ended is when the crawler gave up and quit. And a few weeks before that conference call, Cisco had announced that a RIF of ~6500 employees would begin in mid-August.

The crawler had just been laid off!

Okay, Maybe It Was Just On Vacation

Roughly a week or so went by with no word from the crawler when — wait, what’s that!?

The crawler had came back! Apparently it wasn’t laid off after all! Just when I was starting to feel unimportant and left out, it came back to me.

It did have a new IP address — 173.37.200.9 — so I’m guessing that either it was given a new role as part of Cisco’s restructuring or that it takes Cisco a week to change an IP address (my money’s on the latter).

I decided to shoot an e-mail to Ajit Shah, the guy whose e-mail address appears in the User-agent string that the crawler sends. Although the crawler wasn’t on vacation, apparently Ajit was. I got an out-of-office reply letting me know that he wouldn’t be back around until August 21st, so I assumed I’d hear from him after that.

Apparently Ajit isn’t a very social guy, though, because I never heard back from him. It sorta reminded me of my old co-worker Kevin (hi Kevin) who signed up for Twitter and Facebook and followed people and added some friends, but never himself participated or posted anything (that’s anti-social networking).

Last night, after calculating that Ajit’s crawler had been hitting my website with an average of over 525 requests per day, I decided he just isn’t a very social guy. He apparently wants to read every word I write but can’t even bother to give me the courtesy of a quick little response. I decided just to go ahead and add the IP address to the “don’t ever let anything in from these hosts” access list.

Are They Stalking You Too?

If you’re wondering if Cisco is stalking you too, just check your log files. The exact User-agent string that the crawler provides is:

FAST Enterprise Crawler 6 used by Cisco Systems (aj....@cisco.com)

I’ve “censored” Ajit’s e-mail address there, by the way.

Response from Ajit Shah

If it wasn’t obvious, one of my goals with this article was simply to get a response from Cisco. It worked. Here’s a message I received via e-mail from Ajit Shah:

Yes, Cisco does index prominent community web sites, in order to make it easier for Cisco.com visitors to find blogs, forums and community content. We also do subscribe to services that help us keep up with the many thousands of comments that happen on a daily basis via twitter, blogs and other social forums. The goal is to provide Cisco related information to visitors of Cisco.com.

In addition, see the link from Ben (in the comments below) to Cisco’s search engine. I, for one, was unaware of this resource but am happy to know that it exists. It also quite clearly answers the question in the title of this article.

It appears that several other networking-related blogs, including Jeremy Stretch’s Packet Life, are also indexed in Cisco’s search engine.

{ 6 comments… read them below or add one }

Leave a Comment

Previous post:

Next post: