Computer Forums

Computer Forums (https://www.computerforums.org/forums/)
-   Web Design and Development | Site Reviews (https://www.computerforums.org/forums/web-design-development-site-reviews/)
-   -   Twitter's robots.txt question: (https://www.computerforums.org/forums/web-design-development-site-reviews/twitters-robots-txt-question-207771.html)

Hemat 12-24-2010 01:43 AM

Twitter's robots.txt question:
 
Twitter's robots.txt, It shows everything is disallowed, but surprisingly search engines are crawling and indexing everybody's profiles pages, Why?

JCB 12-24-2010 05:45 AM

Re: Twitter's robots.txt question:
 
i guess the robots.txt file isnt setup corectly?


All times are GMT -5. The time now is 07:25 PM.

Powered by vBulletin® Version 3.8.8 Beta 4
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Search Engine Friendly URLs by vBSEO 3.6.0