Title™: Long Term Data Storage Options.

[Loki, The Fallen]

I come before you with an issue I've been thinking about lately.

With a large amount of data, I am begining to run into the issue of storage, thinking long term while maintaining accessability. I have considered Writable CDs and DVDs, but have read articles about the dye degrading over time, and also have concerns about contamination of said medium.

As I near another bout of Hard Disk purchases, I'm starting to wonder if I've missed other possible mediums. Are there any that you may know of that I have missed?

And barring that, should I remain with my Internal Hard Disk Method, I have another question. I have used Maxtor Hard Drives nearly exclusively for a while now. I have yet to have a failure (Fingers Crossed!), with any of my disks, starting from the PC I built in '99, culminating with the first rackmount PC I assembled back in early '05. This all occured in spite of many reviews and first hand experiance from family members and friends.

With creating a Raid array being a given once I do acquire these newest additions, and SATA II also being required, as I want faster transfer rates then my current ATA provides, does anyone have a brand preference, preferabely with first hand experiance? Although I may be a posterboy for Maxtor, I can only assume I am just freakin' lucky as all hell. I am mostly looking for reliability, noise not being a major factor.

All advice is greatly appreciated.

[bluestarultor]

If you want a lot of storage for a reasonable price, you might want to check into the various cards and other removable drives. Or, failing that, you could always, y'know, delete something. I have an 80GB drive which is largely free because music goes on CDs and I only essentially have the stuff on here that I could ever possibly be using in the forseeable future. As in, I see no reason anyone should ever have a 200TB drive FULL OF FREAKING MUSIC AND MOVIES. You're wasting space if you're keeping the stuff you don't want, don't see yourself wanting anytime soon, but are keeping in case you're wrong about yourself.

[Demetrius]

Just an FYI, Loki is wayyyy into the TB range of anime, etc.

[Mirai Gen]

I think that DVD burning is probably the best way to go. Keep it in data format and you can store a bunch of stuff, and it's totally safe

Except from like, fire.

[Sithdarth]

I hate to be the bearer of bad news but there really is no way to future proof your data storage. As new storage mediums evolve old ones become obsolete and its harder and harder to find ways to retrieve the information. So even if your DVD's were rated to faithfully store uncorrupted data for 20,30, or even 50 years there is a good chance they'll be obsolete in about 10-15. At the very least it will become harder to find ways of retrieving your data.

Even the lowly hard disk is starting to be replaced. In the next few decades we will probably see the birth of a solid state three dimensional data storage system. The long and the sort of this is that you're just going to have to continually migrate your data to new mediums every time a new medium becomes the standard. I wouldn't migrate to Blu-Ray or HD-DVD just yet but sooner or latter you probably will have to migrate to one or the other.

[Eltargrim]

I share the same compunctions about optical media as you do, Loki; I'm working on moving all of my stuff onto magnetic media as well.

Personally, I'm thinking about the "Not a computer, not an eHDD" method. Basically I'd build a bare-bones box, preferably with a ton of SATA ports and 3.5" slots, set up a RAID 5 (For the scale you're thinking of, this is the best option in terms of needed redundancy and space), and buy a multitude of the biggest drive your manufacturer makes. Then construct the array and set it as a network drive, and connect the array box to the network. There's probably an enclosure designed to make this easier, but I have not looked, and this would probably be cheaper and allow for you to customize your parts/have a spare computer around. You might want to have a separate disc for the OS, however; a spare 20GB drive would suffice. Takes up a SATA spot, but hey, with 500GB drives, RAID 5 and 6 ports, you'd still end up with ~2TB of storage.

Now, the manufacturers. Personally, I'm extremely fond of Seagate, as they have a 5-year warranty. Normally, that's not worth very much; your data is just as gone. But with the redundancy of RAID 5, you just pop in the new disc and wait for the reconstruction. If you get to a very large number of discs, however, performance falls (I have no idea what this number is, however; you should be safe).

Now, I know you probably know most of this, and in summary I'll say that I second your RAID idea and suggest RAID 5 with Seagate 500GB drives. But this helped me organize my thoughts, and might just inspire others to try it out themselves.

Finally, unless you're looking to back up data for use 10+ years from now, you should have absolutely no problems with interfacing. I mean, pATA has been around for 21 years, and they're still making new motherboards with it. Even with accelerating rates of standards change, you should be good. Good luck, my fellow pack rat!

[Sithdarth]

You know Eltargrim if you are going to soak that much money into a networked storage solution you might as well by a blade server. If I am remembering right each blade can be set up to act in several RAID configurations. Something like this although its a bit pricey (http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=3217195&CatId=207). Which would give you 16TB of storage and could be set up in a RAID 10 configuration. If you bought something like this (http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=2078821&CatId=209) and filled it with the previously mentioned blades you'd get about 256TB of data storage and a power bill through the roof. (Considering the thing would eat about 12.8 KW having 16 800W power supplies. Plus cooling would be an issue.)

Something like this (http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=2948933&CatId=2672) mounted in something like this (http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=1212030&CatId=209) would probably be a better idea. Not only can you swap out the drives in the blade you can also add more blades later for more storage. In fact you could probably find a similar one without harddrives and just buy 4 1TB drives to go in it. Then later repeat the process to upgrade your storage to a total of 8 blades at 4TB a piece or about 32TB of storage. You really shouldn't have to pay more then say $3000 for a 4TB blade.

The advantage this has over building a barebones box and networking it with your computer is that you don't pay for stuff you don't need. If you build a barebones box its going to need ram, some kind of display adapter so you can set it up, it'll probably have onboard sound, and the CPU is going to be vastly more powerful then it needs to be. After all that it both takes up more room and is less upgradeable then just running a rudimentary blade server.

Of course you could always buy something like this and be done with it. (http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=3142388&CatId=2672)

Edit: Can you tell I'm having a few storage issues of my own?

[Eltargrim]

Oh yes, yes I can.

I suppose it really depends on the scale of your need. 2TB would be plenty for me, and the solution I set up would also give me a spare computer, which is something I'll never say no to. If Loki is already well into the TB range, then it probably wouldn't be enough.

As to the issue of RAID, I prefer arrays with parity rather than mirroring, if only because you lose a smaller percentage of your total space to redundancy. RAID 5 is also easier to expand than RAID 10, which is another stroke in it's favour.

[Sithdarth]

Well any of those system can run in basically any RAID format you want. I prefer blade server solutions myself for a few reasons.

1) Scalability; I can always add more almost without limit. If one blade fills up a buy a new one and slap it under the old one. That is until the rack fills up but then I can just buy a new rack.

2) Modular Elements; Almost like scalability except that it means I can purchase the system piece by piece and still have a functioning system in between. This applies not only to the blades but the Hard Drives in the blades as well.

3) Upgrade-ability; the twin sister of modular elements. I can buy better Hard Disks later and just stick them into the older blades. (To a point.)

4) Footprint; A blade even a stack of blades takes up far less space per TB of data than a whole other computer would. I mean you can get like maybe 6 Hard Disks into a case and that's really pushing it. I can easily buy a blade that holds 4 and stack 3 or 4 of these blades in the exact same area that the barebones box occupies. That's somewhere between 200 and 267% more space per unit of area.

5) Efficiency; you aren't paying for a parts to a whole other computer you may never actually use. Further you aren't paying to continuously run a whole other computer meaning blades generally use less energy per TB of data. (Although some of the bigger blades are mini-computers in themselves)

6) Ease; They're darn near plug in play if you by the fully assembled ones with Hard Drives included. Not to mention the drives can be set up to appear to your computer as if they were actually internal drives. A set up like that is much harder to achieve with two networked computers.

Edit: Actually go with this. (http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=3143432&CatId=2778) Its got basically all the benefits of a blade plus it is fully automated. Not to mention cheap.

[Nikose Tyris]

Two things:

External Hard Drive.

Fireproof Safe.


I have all of my writing in one, at my mothers house in the back of the closet. I know the fucker still works.

[Loki, The Fallen]

I've pretty much started with that idea, Sithdarth. Managed to acquire a (est.) 24U cabinet from work, and have built a barebones computer kinda like Eltargrim's idea, and populated it with ATA/133 drives. I managed to find a nice box on EBay with a bunch of hot swappable slots for 3.5 HDDs. It's worked well for a while, but due to a lack of forethought on my part, I have hit the population limit for it. (Lack of free ATA ports, Lack of PCI slots for more cards, etc)

I like the idea of those blades, and they appear to be a lot easier to set up then the box I created in the past. But that price tag really hurts, especially the 16TB unit. As current usage rates are pointing to around 2-3TB a year, it would take a significant project to actually take off before that would be in my budget. I was able to build the box I currently run for significantly cheaper, although 4U for primarily 2-3 TB of storage does seem to be a waste. I may be more likely to go with this (http://www.tigerdirect.com/applications/SearchTools/item-details.asp?EdpNo=2948932&CatId=2672), as it A) is cheaper, less "Sticker-Shock" B) I can start with whichever drive I like. I may have missed it on the site, but are there size limitations for HDDs on those blades?

Maybe that is the route I should go. If I hadn't already acquired the rack and started buying equipment designed for it, I would be tempted to try that 'Robot' idea, that looked interesting, but I foresee a large pile of small boxes sitting in the corner. Its the same issue I have with eHDDs, I can't see a way to store them as nicely/organized as a rack.

I guess I'm kind of torn though. Redesigning the 4U unit I have would be a good idea, but most of the larger cards with more ATA ports I priced out are near $400 or so, (Since the bays are all built for ATA/133, it kinda kills the idea of going SATA...), so that would be quite the hit to take, in addition to buying more drives, and a new MB that has the PCI slots available to make that work. But rethinking the 1U blade idea (barebones), it seems slightly cheaper in comparison, with only around $800 for the unit, then adding HDDs.

I didn't think much about warranties in the past, usually by the time anything I purchase dies, the item is well out of warranty, but a fair amount can happen in 5 years and going with that Raid config could make that option useful. Perhaps that would be the idea to go with.

I saw a mention of a cooling issue for this set up, I have already assumed I'd have to deal with room climate control, will I really need to look into more cooling in addition to regular air cooling?

[Sithdarth]

Here is the product website form the company that slaps their name on that blade. (http://www.linksys.com/servlet/Satellite?c=L_Product_C1&childpagename=US%2FLayout&cid=1115416945617&pagename=Linksys%2FCommon%2FVisitorWrapper&lid=4561701824L08)

And here is there list of recommended drives. (http://www.linksys.com/servlet/Satellite?blobcol=urldata&blobheadername1=Content-Type&blobheadername2=Content-Disposition&blobheadervalue1=application%2Fpdf&blobheadervalue2=inline%3B+filename%3DAVL%2BDrive% 2BList%2BFEB%2B2007.pdf&blobkey=id&blobtable=MungoBlobs&blobwhere=1130868020645&ssbinary=true&lid=3045161927B22) It seems with recommended drives these blades top out at 3TB though you could probably swap in 1TB versions of drives from the recommended manufactures and be just fine.

With a bunch of 1U low power blades running on a LINUX OS built right into the blade itself you shouldn't have to much in the way of heating problems. Though that does depend on just how many you pack into your rack. As well as were you place the rack and where you place the blades in the rack. You want as much open space around the rack as possible and as much open space between blades as possible. You'll probably also want an air conditioner in the room with it. If at all possible I'd suggest you also seal the door as air tight as possible as well as any windows. You also might want to upgrade the filters in the air ducts leading into the room to HEPA.

I know that seems like a bit of over protection but that's how big companies set up their server rooms. If you are going to shell out that much cash for hardware you might as well make sure its running in as near perfect conditions as possible. You'll probably want to set the air conditioner to keep the room below 70-75F just to be safe. The natural action of the air conditioner will keep the room dry and the filters and sealing of windows and doors will help keep the dust down. (Dust is a major problem for desktops and you really don't want to have to take apart a 10+ blade server every 3-6 months to dust out all the blades.)

If you are really paranoid you might place a thin metal strip right in front of the door to the room and run a wire from that to the ground of your house. That way each time you step on it before you enter the room you'll drop any static charge you might have picked up. This isn't normally terribly important but you are entering a rather cool very dry room. These are ideal conditions for a very significant static spark. You may also give thought to grounding your rack. Its going to be in that dry, cool environment with air continually moving over it. That's a prescription for a very large static charge in an environment where that charge could jump almost half a foot. That could be bad for both you and your blade server even though it is less likely the plain old heat damage.

Edit: Would you believe the only computer oriented classes I have ever taken are Keyboarding and C++ programing? This all comes from a really healthy side interest in all things electronic. Maybe I should have listened to the IT teacher in high school and gone into server management or something.