Originally Posted by j03
This makes me wonder how huge datacenter's back-up their data... Do they do it off, or on site? Or a combination of both? It's a huge problem!
one of the companies that we support have roughly 4TB of data. that is backed up in the following way.
differential backups to tap (2 lto4 tapes) on a nightly basis (kept for one week)
full backups made on fridays, (4 LTO4 tapes) (job runs till sunday) to backup all data (Kept for 1 month)
full backups made on the last friday of the month -kept for a year
Full backups made on the last friday of the year -kept forever.
the tapes are all loaded into a robotic loader, so there is 1 tape drive, the tapes are put into a magazine and inserted into the robot and the robot picker picks up the tapes and loads it into the drive, (it decides the tape to load either by it's position in the magazine or by a bar code printed onto labels on each tape.
for another customer where we backup a lot more data we have a robotic loader that has two tape drives in it, another tape loader has four tape drives in it (that is a very large tape library that can have 24 tapes loaded at once.
but as I said before these tapes are expensive, the tape loaders and libraries are even more expensive and there is a finite throughput that can be written to the drive, (limited by the bus speed of the tape library connection bus, and the network that's going to the server that is controlling the jobs etc.
(if you allow the hours of between 6pm and 8am) you can practically backup on a single tape drive around 2TB. -that also allows time for the software or hardware to compress the data so that it actually fits onto a single drive.
(assuming that your media server is one device and that the tape drive is attached by SCSI to the one server).
to backup 37TB you can do a few things, but these are going to be limited.
step 1 would be to get an intelligent storage device that can control the tape library, that would take out a bottle neck at the network stage.
after this you're going to need to start looking at multiple tape drives.
So far as who backs up the data, the daily, weekly, monthly yearly tapes etc are collected from our data centres on a daily basis by a third party called Iron Mountain, they take the tapes away and store them in secure facilities. and deliver the next days tapes.
Practically, if you can throw enough resource at a backup problem then you can solve most through intelligent design and using the right equipment...
of course, the OPs backup requirements are 37TB.
there are 37 machines with 1TB drives,
the drives aren't going to be full,
there will be an amount, (probably at least 10GB) on each machine that is used by windows installation and program files.
there will undoubtedly be personal data that doesn't need to be backed up.
and even if the drives were absolutely full you need to figure out how much of the data is duplicated.
how many common project files are there that every one of the 37 computers each have, but you only need 1 backup copy of?
(this is why I said that they need to rationalise and actually think about what needs to be backed up).