Twitter's robots.txt question:

Hemat · 12-24-2010, 12:43 AM

Twitter's robots.txt, It shows everything is disallowed, but surprisingly search engines are crawling and indexing everybody's profiles pages, Why?

**JCB** · 12-24-2010, 04:45 AM

i guess the robots.txt file isnt setup corectly?

12-24-2010, 12:43 AM	#1
Hemat Beta Member Join Date: Dec 2010 Posts: 1	Twitter's robots.txt question: Twitter's robots.txt, It shows everything is disallowed, but surprisingly search engines are crawling and indexing everybody's profiles pages, Why?

12-24-2010, 04:45 AM	#2
JCB Site Team Join Date: Oct 2004 Posts: 1,294	Re: Twitter's robots.txt question: i guess the robots.txt file isnt setup corectly? __________________ My Info :Ex-Owner of Computer Forums .org -- CCA in Citrix Presentation Server 4 + 5 + Xenserver 5.5 Site Info : Free Boot Disk Download - Latest Articles - Latest Posts - Site Rules More Info : If you have any Questions Send Me A PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode