| Author |
Message |
ewielenga
Joined: 09 Oct 2006
Posts: 25
Rank: 1
|
 Posted: 09 Oct 2006 11:34 Post subject: prevent crawling of freesites with robots txt |
 |
|
OK,
I have a guy submitting that on his root robots.txt has the following:
User-agent: *
Disallow: /gall/
Disallow: /gall1/
Disallow: /gall2/
Disallow: /gall3/
Disallow: /gall4/
Disallow: /gall5/
Disallow: /gall6/
Disallow: /gall7/
Disallow: /gall8/
Disallow: /gall9/
Disallow: /gall10/
Disallow: /gall11/
Disallow: /gall12/
Disallow: /gall13/
Disallow: /cgi-bin/
Disallow: /img/
domain: soccerwank.com (also sexcarrot.com with different directory names his freesites are in)
On the freesites themselves, in the head, is the meta:
meta name="robots" content="index, follow"
I was running a link checker and was getting flags with this message:
"The link was not checked due to robots exclusion rules. Check the link manually."
Hence me looking at the root robots file.
Seems very fishy to me, and this is titled 'possible cheaters' but I can't fathom whether this is an honest mistake, as obviously his freesites aren't going to get pickjed up by the SES, or just a way to glean traffic from LLs. |
|
| Back to top |
|
OffMan
Joined: 06 Oct 2006
Posts: 73
Rank: 0
|
| Posted: 09 Oct 2006 11:35 Post subject: |
|
|
| Also looks like a way to turn recip links (A->B->A) into one-way links from the LLs to his domains. (One-way links being more valuable.) |
|
| Back to top |
|
Porny
Joined: 09 Oct 2006
Posts: 25
Rank: 0
|
| Posted: 09 Oct 2006 11:40 Post subject: |
|
|
that is the thing that bugs me...why have that in the meta unless he forgot to take it out from a cut & paste off something else. Kinda tricky imo.
he's a member of the board, maybe we'll hear something. |
|
| Back to top |
|
JohnLev
Joined: 06 Oct 2006
Posts: 48
Rank: 5
|
| Posted: 09 Oct 2006 11:41 Post subject: |
|
|
| This isn't the only forum member doing this. Like Edith I'm not sure it's exactly cheating so I would like to hear more opinions on this. |
|
| Back to top |
|
Porny
Joined: 09 Oct 2006
Posts: 25
Rank: 0
|
| Posted: 09 Oct 2006 11:42 Post subject: |
|
|
| yes me too. I wouldn't consider it cheating, really - it's not breaking any rules, but it's misleading. |
|
| Back to top |
|
virgynews
Joined: 09 Oct 2006
Posts: 17
Rank: 0
|
| Posted: 09 Oct 2006 11:45 Post subject: |
|
|
If someone's going to take the extra time and energy to stuff their robots.txt file in an attempt to get more back then they're giving I would consider that following the rules, but still acting with malicious intent.
Malicious intent falls under the unspoken rule of, "I don't like your business practices, therefore, I don't want to do business with you."
In this situation, I probably wouldn't send a rejection email, or even ask what's up. I'd just silently make their sites dissappear with a quick click of the delete button.
Jel, thanks for bringing this issue up. As if I didn't already have enough to check for... |
|
| Back to top |
|
virgynews
Joined: 09 Oct 2006
Posts: 17
Rank: 0
|
| Posted: 09 Oct 2006 11:46 Post subject: |
|
|
Methinks folks don't look in this section often enough.
I think I'll send TT a note and tell him to look here; I like him and it surprises me that this kind of thing would be done on purpose. Maybe he's got a good explanation for it.
Either way, it certainly isn't in anyone's LL rules that it can't be done, so...? Weird situation. |
|
| Back to top |
|
|
|
|
|
|
|
|
|
