Comments on robots.txt File - AddedBytes.com

Comment on robots.txt File

2008-06-13T22:29:10+01:00

Comment by johan ( http://telefonix.co.uk )

caren, google ".htaccess rippers". It's another file you put in your root, it stops bad boys ripping off your site.

Comment on robots.txt File

2008-04-07T06:02:30+01:00

Comment by Brent Lagerman ( http://www.mimoymima.com )

first off great site, I've stumbled upon you many times in the past and have always found great info here.

secondly can you help me with this problem:

I'm trying to disallow a directory /clients/ but allow only one file in the directory /clients/index.php

is this possible with

allow /clients/index.php
disallow /clients

does it matter the order or will this just not work at all.

brent
@
mimoymima.com

Comment on robots.txt File

2008-02-03T00:20:04+00:00

Comment by Caren ( )

I have had people stealing from and downloading my entire website. I have exact copies of it with a few minor changes and them claiming it as thier own. I have put a great amount of time and money into creating it. I am looking for something to lock the pages or encrpt them so they can not be stolen. I understand that if you encrpt the html with a program that your page or website can not be crawled or index. Is this true? Any suggestions on how to protect my work and stil be indexed so my site can be seen? Are there any programs that you would suggest? Thanks!

Comment on robots.txt File

2007-12-06T23:09:28+00:00

Comment by Julian ( http://www.toporganicdiet.com )

I instructed robot.txt to disallow indexing my website, but google still picked up and indexed all the pages even src and images forbidden directories. Did anybody have the same experience?

Comment on robots.txt File

2007-10-27T15:53:14+01:00

Comment by Anonymous ( )

can i know where to put robot.txt?

Comment on robots.txt File

2007-10-18T10:06:03+01:00

Comment by James Ryddel ( )

Fix that link Dave.

http://www.ilovejackdaniels.com/seo/robots-txt-file/

That's a free tip :0)

Comment on robots.txt File

2007-09-24T22:08:00+01:00

Comment by Haris khan ( )

This article really helps me in writing robots.txt file.

Comment on robots.txt File

2006-10-31T14:47:06+00:00

Comment by Commentar ( )

Mohammad,
There is one sure fire way to "come in first page of google search engine when a person search with a keyword of OUTSOURCE".

* * * BUY THE KEYWORD * * *

Comment on robots.txt File

2006-10-19T21:33:51+01:00

Comment by mohammad ali ( http://www.outsourceb2b.com )

please suggest me i m developing this company site and i m nil in kn owledge of SEO but i know what is seo?but this company site to come in first page of google search engine when a person search with a keyword of OUTSOURCE and then it should come in first page please give suggestion in this how i have to proceed in design so that they can reach that position.

Comment on robots.txt File

2006-09-11T23:02:09+01:00

Comment by Sean ( )

Good question, Ulu.

Also just a heads-up ILJD, the link to the Robots.txt validator is broken, or rather it leads to a redicrect which leads to the webmasterworld donate (login?) page.

Love your site, good article.