.htaccess and robots.txt

Discussion in 'Site Security & Legal Issues' started by finkey, Dec 8, 2005.

  1. finkey

    finkey Aspirant

    12
    0
    +0
    I want to stop prying eyes from looking at my robots.txt like this: http://www.foo.com/robots.txt. It could give them clues about restricted directories. The less they know, the better. If I use .htaccess, will it interfere with robots' ability to crawl as directed around my site? Any better solution?
     
    1
  2. DChapman

    DChapman Devotee

    2,878
    730
    +13
    Do you want the directories to be open to everyone? Just not spiderable? Or can access to the specific directories be locked down?

    I would very strongly recommend you not attempt to lock down your robots.txt with .htaccess
     
  3. finkey

    finkey Aspirant

    12
    0
    +0
    The particular sub directories will be password protected with .htaccess. Your answer points to not use file protection on robots.txt. The directory for robots.txt is my home directory and has to be open to everyone. So, there's no way to stop snooping on the robots.txt? Thanks!
     
  4. Moparx

    Moparx GNU/Linux Inside

    1,352
    620
    +2
    there is no way to stop anyone from looking at your robots.txt.

    if the subdirectories listed are password protected already why dont you just remove the listings from the file?
     
  5. DChapman

    DChapman Devotee

    2,878
    730
    +13
    Not in a manner that wouldn't hurt you.

    If your directories are password protected, just don't list them on robots.txt
     
  6. finkey

    finkey Aspirant

    12
    0
    +0
    Part of the concern was could robots harvest shopping cart records. I don't plan on allowing customers access to that anyway on secure server. They'll have to print out their invoice at point of sale. I will delete the online record after a day or so.

    Thanks for the solution and assist.
     
Verification:
Draft saved Draft deleted