Scribd Logo
Upload

Welcome to Scribd!

  • Robots

    Uploaded by

    avanti zone
    15 views1 page
    This robots.txt file allows all major search engine crawlers like Googlebot, MSNBot, and Yahoo to access the entire site but blocks the /cgi-bin/ directory. It also allows most other common bots and crawlers to access the entire site.

    Original Description:

    Original Title

    robots (1)

    Copyright

    © © All Rights Reserved

    Available Formats

    TXT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    This robots.txt file allows all major search engine crawlers like Googlebot, MSNBot, and Yahoo to access the entire site but blocks the /cgi-bin/ directory. It also allows most other common bots and crawlers to access the entire site.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    15 views1 page

    Robots

    Uploaded by

    avanti zone
    This robots.txt file allows all major search engine crawlers like Googlebot, MSNBot, and Yahoo to access the entire site but blocks the /cgi-bin/ directory. It also allows most other common bots and crawlers to access the entire site.

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 1

    robots.txt generated by www.seoptimer.

    com
    User-agent: Googlebot
    Disallow:
    User-agent: googlebot-image
    Disallow:
    User-agent: googlebot-mobile
    Disallow:
    User-agent: MSNBot
    Disallow:
    User-agent: Slurp
    Disallow:
    User-agent: Teoma
    Disallow:
    User-agent: Gigabot
    Disallow:
    User-agent: Robozilla
    Disallow:
    User-agent: Nutch
    Disallow: /
    User-agent: ia_archiver
    Disallow:
    User-agent: baiduspider
    Disallow:
    User-agent: naverbot
    Disallow:
    User-agent: yeti
    Disallow:
    User-agent: yahoo-mmcrawler
    Disallow:
    User-agent: psbot
    Disallow:
    User-agent: yahoo-blogs/v3.9
    Disallow:
    User-agent: *
    Disallow:
    Disallow: /cgi-bin/

    You might also like