While on the phone with Jerry West of Web Marketing Now, he told me that, “Having a robots.txt file in the root directory of your web site greatly increases a it’s spiderability.” According to his tests, “Significantly more pages get spidered, especially when the pages are two or three levels deep.” Hmmm, I wonder if search engine positioning goes up as well?

I’ve never used robots txt files on any of my sites, so I started asking around some of my SEO friends and turns out most of them use the same software to make their Robots file. The one they recommend the most is the professional version of Robot Manager.

What’s cool about this package, is that it not only generates foolproof Robots txt files, it tracks all the spidering activity that goes on at your site. You’ll know exactly which spider came, and when, and what it got. You can also use it to exclude malicious bots from your site, so it’s not busy serving some evil bot when the good Google bots are trying to get at your pages.

After some of my friends and I ran our own tests on larger sites (being a mininet fan I don’t have many large sites . I found an average of 11% more pages got indexed and on the SERPS (search engine results pages) my positioning went up an average of three places. The only change I made to the site was to add the Robots txt file to the root directory of my server (the same place the index page is stored).

According to Jerry’s private newsletter, my results are actually a little low. He’s been getting much bigger numbers. He’s been getting over 20% more pages spidered than before. But I expect that its due to his sites being quite a bit larger than mine.

So the proof is in the pudding as they say. And it’s hard to disagree with success. Point being, get a robots.txt file on your site. Either make it yourself or go buy software to do it for you. Your search engine positioning and the amount of pages you get spidered depends on it.

by Michael Campbell

Author of….

Revenge of the Mininet… Advanced search engine linking strategies and diagrams for increased revenue.

Clickin’ it Rich… The complete work from home business training system for new affiliates.

Nothing but ‘Net… Simple internet marketing strategy that made $750,000 in less than a year.

Internet Marketing Secrets Newsletter… Learn how to harness the money making power of multiple internet revenue streams like search engines, affiliate programs, paid advertising, opt-in email, newsletters, ebooks and lot more in internet marketing secrets. Subscribe now and get a free lifetime membership.