hi all, I know that most of you reading this either love to bag on Bing or you’re only familiar with hearing bad things about Bing, but if you haven’t taken a look at their Webmaster tools. Even if you hate Bing for whatever reason, their Webmaster tools are pretty awesome. I think there’s something to be said for them reaching out to their adopters to obtain feedback in this manner. And for as much as I like Google, I like the prospect of competition for Google even more and I see Bing in a prime position to do just that. But I digress.
Hmmm Google has had the ability to set crawl frequency in Webmaster Tools forever. But why in the hell would you want to? The search engine should be able to determine a good crawl frequency for your site's scheduled deep crawl based on the amount of new content they find each time they return to the site. That is what Google does. Of course, they will crawl your site in between its schedule crawl times because they find links to your sites on other sites when they are performing those site's scheduled deep crawls. If each time the crawler returns they find bunches of new pages then they will up the crawl rate. They will continue to do so until they find a balance between your site's content generation rate and their crawl rate. It's stupid IMO to try to "control" your site's crawl rate. You're better off just letting it happen naturally. The only time it makes sense to try to control it with meta tags, robots.txt, or webmaster tools is if your site has some crappy web host and crawling your site brings the server to its knees. And in that case... the solution is NOT to set crawl rates manually... It's to find a new web host.
