Results 1 to 2 of 2

Thread: What is robots.txt file?

  1. #1
    Join Date
    May 2009

    What is robots.txt file?

    This is a text file used to control the bots of various search engines to access the files and pages present at your site.It is quite useful in search engine optimization.
    Its normally used to stop the search engine bots from accessing or indexing certain pages or stopping few of search engines.

    if you dont want to stop any bot then place an empty robots.txt file at the root directory of your site or just write User-agent: * in it.

    For e.g :

    and if u want to stop someone then you can inlclude the name of that bot which u want to stop accessing these files.

    the synatx which you have to write in order to block the bot is

    Disallow: /nameofwebsite/memberlist.php

    replace nameofwebsite with your name

  2. #2
    Join Date
    May 2009

    Re: What is robots.txt file?

    so at last forum is alive :P

Similar Threads

  1. [WTS] PR1 File Mirror Site - Upload to 15 File Hosts
    By ChaoscripT in forum Sites
    Replies: 0
    Last Post: 26 September, 2010, 10:53 AM
  2. Help with Robots.txt
    By 5starpix in forum Building
    Replies: 4
    Last Post: 11 February, 2010, 03:27 AM
  3. robots.txt help
    By Sami4u in forum Building
    Replies: 9
    Last Post: 27 September, 2009, 07:43 AM
  4. Robots.txt?
    By dmi in forum Managing
    Replies: 15
    Last Post: 9 September, 2009, 17:15 PM
  5. Block Robots and Web Downloaders with robots.txt
    By Will.Spencer in forum Managing
    Replies: 12
    Last Post: 6 June, 2009, 15:40 PM

Tags for this Thread


Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts