Scribd Logo
Upload

Welcome to Scribd!

  • Robots

    Uploaded by

    Biswadeep Sarkar
    8 views1 page

    Original Title

    robots.txt

    Copyright

    © © All Rights Reserved

    Available Formats

    TXT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    8 views1 page

    Robots

    Uploaded by

    Biswadeep Sarkar

    Copyright:

    © All Rights Reserved
    Download as TXT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    of 1

    # robots.txt generated by smallseotools.

    com
    User-agent: Robozilla
    Disallow: /
    User-agent: Nutch
    Disallow: /
    User-agent: baiduspider
    Disallow: /
    User-agent: naverbot
    Disallow: /
    User-agent: yeti
    Disallow: /
    User-agent: psbot
    Disallow: /
    User-agent: *
    Disallow:
    Crawl-delay: 10
    Disallow: /cgi-bin/
    Sitemap: https://www.reshamshilpi.in/sitemap.xml

    You might also like