Make a search in Google.com for 'robots.txt'.
the 6th result on the first page is : http://www.whitehouse.gov/robots.txt which reads the following
The keyword 'robots.txt' has a global monthly search volume of 246,000.User-agent: *
Isn't this like spamming of the Serps, why doesn't google do something about this. I thought robots.txt was for bots and not for humans.
Anyways, what will a noob user who has no idea and wants to learn about Robots.txt presume about it from this?
@mods : please move if its in the wrong category, couldn't find a more suitable one.
Last edited by Snak3; 3 November, 2009 at 16:40 PM. Reason: typo : spelt robots.txt as roots.txt lol :D
The Secret Definitive Guide To Finding Tons of PR4-PR8 Backlinks with ease <== Boost Your Website Rankings Overnight