Go Back   Computer Forums > General Computing > Server Administration
Click Here to Login
Join Computer forums Today


Reply
 
Thread Tools Search this Thread Display Modes
 
Old 12-17-2011, 05:19 PM   #1
Solid State Member
 
Join Date: Dec 2011
Location: USA
Posts: 12
Default Website crawlers?

what the best website crawler or spider to use to copy a hole site offline i what copy backup so if there shut down site i whatn't lose info and have to go find or pay for it most of the one i have found are trail verison no free one that are up to date help please
__________________

HELLFIRE452 is offline   Reply With Quote
Old 12-17-2011, 07:18 PM   #2
Wizard of Wires
 
setishock's Avatar
 
Join Date: Feb 2005
Location: Not sure
Posts: 10,030
Default

If it's your site simply doing a database back up and a home directory backup will be the solution.
As for sending a bot to someone else's site. Well let's just say that's not real bright. You can be back traced. It could get serious.
Just a word of advice. Web site owners can be a testy bunch. I'd think twice before you did that...
__________________

setishock is offline   Reply With Quote
Old 12-19-2011, 03:38 AM   #3
Solid State Member
 
Join Date: Dec 2011
Location: USA
Posts: 12
Default

no not that i just what a copy of a hole website just so if they shut down in a year or
two i whatn't be lose any info i got off the site
HELLFIRE452 is offline   Reply With Quote
Old 12-19-2011, 03:42 AM   #4
Solid State Member
 
Join Date: Dec 2011
Location: USA
Posts: 12
Default

just trying to find a program or way to copy all that on a website i found this how to site a couple years ago and it was all in txt format and had a zip or rar download or all the file on the site and i been try to see if there some way or program for that i wood love have a copy of fanfiction.com or some there site i go to just in case if they
shut down
HELLFIRE452 is offline   Reply With Quote
Old 12-20-2011, 05:59 AM   #5
Site Team
 
root's Avatar
 
Join Date: Mar 2004
Posts: 8,004
Default

I used to use a program called websnake.
by a company called annawave.

I don't know if it's still being made, but that does exactly what you wanted.
__________________
I didn’t fight my way to the top of the food chain to be a vegetarian…
Im sick of people saying 'dont waste paper'. If trees wanted to live, they'd all carry guns.
"The inherent vice of capitalism is the unequal sharing of blessings; The inherent vice of socialism is the equal sharing of miseries."
root is offline   Reply With Quote
Old 12-20-2011, 03:25 PM   #6
Solid State Member
 
Join Date: Dec 2011
Location: USA
Posts: 12
Default

look and can't find it but thanks for help i been traying to find a freeware verison of this software or
program like and no one what to help or point me to info like this i wood like find a freeware but if
i have to buy one i what to buy one that will setup the same setup as it is online all link together you click on storie and can go chapter to chapter or go to the author page and read up on them and and click one of the storie and go and read them that what i wood like can any tell be witch one of the buy
programs are the best and fast at downloading?
HELLFIRE452 is offline   Reply With Quote
Old 12-22-2011, 05:24 AM   #7
Site Team
 
root's Avatar
 
Join Date: Mar 2004
Posts: 8,004
Default

Google

paid version is top link. (it's worth the $30 to do what you want?)
4th and 5th links are not paid for links.

not that I'm promoting 'that' kind of thing obviously.
__________________
I didn’t fight my way to the top of the food chain to be a vegetarian…
Im sick of people saying 'dont waste paper'. If trees wanted to live, they'd all carry guns.
"The inherent vice of capitalism is the unequal sharing of blessings; The inherent vice of socialism is the equal sharing of miseries."
root is offline   Reply With Quote
Old 12-25-2011, 06:32 PM   #8
Solid State Member
 
Join Date: Dec 2011
Location: USA
Posts: 12
Default Re: Website crawlers?

than what pay for one wood be the best? becuase i what to save it like save a web page and than it save all linking pages to one i save so the hole web site will copyed i read fanfiction stories all the time i wood love for it to work so that i only have to download one time and the programs download all the chapter after that? and thanks for the help
HELLFIRE452 is offline   Reply With Quote
Old 12-26-2011, 07:44 PM   #9
Site Team
 
root's Avatar
 
Join Date: Mar 2004
Posts: 8,004
Default Re: Website crawlers?

that mirrors a whole site, it does it by saving the text and pictures of the site that you're on to your local computer.

also you set a link depth, so it follows links on the first page to pages inside the site.

the tool os meant for mirroring whole websites.

it is exactly what you want to copy a whole website off line. anything that is a link that you can click the program will follow and mirror the content at the end of that link.
__________________
I didn’t fight my way to the top of the food chain to be a vegetarian…
Im sick of people saying 'dont waste paper'. If trees wanted to live, they'd all carry guns.
"The inherent vice of capitalism is the unequal sharing of blessings; The inherent vice of socialism is the equal sharing of miseries."
root is offline   Reply With Quote
Old 01-14-2012, 12:36 AM   #10
Wizard of Wires
 
setishock's Avatar
 
Join Date: Feb 2005
Location: Not sure
Posts: 10,030
Default Re: Website crawlers?

When your bot starts sucking down massive amounts of bandwidth you're going to get either the host's attention or the webmaster's. Either way the bot's gonna get blocked.
__________________

setishock is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are Off



All times are GMT -5. The time now is 02:21 PM.


Powered by vBulletin® Version 3.8.8 Beta 4
Copyright ©2000 - 2017, vBulletin Solutions, Inc.
Search Engine Friendly URLs by vBSEO 3.6.0