Go Back   Computer Forums > Web Design | Website Development > Web Design and Development | Site Reviews
Join Computer forums Today

Thread Tools Search this Thread Display Modes
Old 12-24-2010, 12:43 AM   #1
Beta Member
Join Date: Dec 2010
Posts: 1
Default Twitter's robots.txt question:

Twitter's robots.txt, It shows everything is disallowed, but surprisingly search engines are crawling and indexing everybody's profiles pages, Why?

Hemat is offline   Reply With Quote
Old 12-24-2010, 04:45 AM   #2
Site Team
JCB's Avatar
Join Date: Oct 2004
Posts: 1,292
Default Re: Twitter's robots.txt question:

i guess the robots.txt file isnt setup corectly?
My Info :Ex-Owner of Computer Forums .org -- CCA in Citrix Presentation Server 4 + 5 + Xenserver 5.5
Site Info
: Free System Recovery Disk - Latest Articles - Latest Posts - Site Rules
More Info
: If you have any Questions Send Me A PM
JCB is offline   Reply With Quote

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off

All times are GMT -5. The time now is 06:34 AM.

Powered by vBulletin® Version 3.8.8 Beta 4
Copyright ©2000 - 2016, vBulletin Solutions, Inc.
Search Engine Friendly URLs by vBSEO 3.6.0