Yahoo last made an update in early March and it did not shake up the net too badly. But that could have just been the beginning of this major update by Yahoo!. Yahoo! has created a new spider bot (crawler) named Slurp 3.0. Over the course of the next couple weeks they will be implementing these new crawlers as well as some infrastructure (software) updates, like when Google introduced Big Daddy and the Mozilla crawl bots. Here is some technical changes yahoo said will be introduced with this spider:

“a) The crawlers will start crawling from a different and much smaller set of IP addresses, but it’ll still be from the Any reverse DNS checks to identify our crawler will continue to work. Please note that if you’re using IP-based recognition of our crawlers, you might see a drop in crawl/coverage from Yahoo! We strongly recommend that you move to reverse DNS-based identification of Yahoo! Slurp if you’re using any other method to avoid this problem. The current set of IPs will disappear from your web logs in the next several weeks.

b) The crawlers will also publish a new user-agent, ‘Yahoo! Slurp/3.0.’ Existing robots.txt directives for ‘Slurp’ or ‘Yahoo! Slurp’ will continue to work, but if you have directives specific to ‘Slurp/2.0,’ they won’t be recognized by the new crawler (though usage of the ‘Slurp/2.0′ user-agent is very rare on the web, so you won’t likely be affected). We recommend specifying the shorter version of: User-agent: Slurp. Check out “How do I prevent my site or certain subdirectories from being crawled?” on our Help page for more details.” ……… Read Full Story Here

It looks as if this update is going to cause some havoc in its wake for webmasters everywhere. This is always to be expected though when an update this major takes place. We have not seen an update from Yahoo! this big in a long time.

One poster over at searchenginejournal.com is claiming the new spider is caching pages at a speed faster than that even of Google. He also mentions that it is caching sites from links that even have a no follow tag included. ….. Read More here

The news of this update was just released to the general population by Yahoo Yesterday, April 14, 2008, So not a lot of information yet on how this can really effect your website but a good forum to watch is over at Webmaster World where members are posting the effects on their websites in various niches. ………………. Go to the Webmaster World Forum here

As more information comes out about Yahoo! Slurp 3.0 we will let you know, to help keep you up to date and on top of your rankings. Remember Search Engine Marketing is the Bread and Butter of your site so do not get lost in these updates check back here, where we know SEO!

Jim Trivolette

“The SEO Doctor”