webcrawlers

[allat]

Would anybody be able to tell me how you tell when there's a webcrawler?
How can one prevent it?

Thanks.

[Comp_Pro]

You can prevent them by installing an anti-virus, anti-spyware and firewall. A good combination would be AVG free anti-virus, Zonealarm Free Firewall, and SpyBot - Search and Destroy.

Be sure to run regular scans (at least once a week) to remove any that have made their way onto your computer. If you really want to know if you have one at the moment, just install SpyBot - Search and Destroy and update your definitions and scan your computer with it. If there are any botnets/webcrawlers on your computer, it will delete it.

[root]

web crawlers are what search engines use to index websites.

they literally crawl the internet following links, reading pages and reporting the content back to their providers, that's how Google manage to index the internet.

you can (supposedly) stop web crawls by using a file on the root of your web server called robots.txt, or assuming the web crawlers don't pay attention to this you can regularly review your web server logs and ban traffic from web crawlers IP addresses or address ranges.


if you mean worms, then firewalls can stop those.

[Comp_Pro]

Opps, excuse my mistake. I thought webcrawlers were botnets