Title says it all.
ITT we discuss data storage >12 TB
Classical Raid Levels, ZFS, ....
Feel free to Post your Storage Rigs, Specs,...
Ask questions....
Discuss Builds,...
I just upload all my videos to Youtube. Done.
raid is not for backups
>>52048446
why do people call it RAID 0?
that doesn't make sense
>>52048446
I've got 20TB of HDD's all in my mediabox. Has been on 24/7 since 2009 and working without a hitch.
Unfortunately I'm using JBOD, because I don't have the money for a proper raid setup. I can't just throw all my current drives in a raid and expect them to work. They are of different sizes/manufacturers.
Filesystem is NTFS, because im too lazy to setup a linux server for accessing my files.
drives are mostly HGST's 3TB's.
>>52048498
Why not?
>>52048918
i think he meant this:
if you have a RAID in your PC, and you store your files on that RAID, some people think that it's backed up. (and as such protected against data loss)
The truth is that a real backup is a COPY of your existing files (unlike RAID1). and these copies provide prevention of dataloss in case of accidental deletion, or malware.
a RAID only provides prevention of dataloss in case of a single drive failure. if you have malware (cryptolocker) or accidentaly delete files, these files are sadly lost, and thus not backed up.
sure, you can combine the two and have a backup on a RAID for performance increase.
My raid0 died.
All my movies standing by for backup all lost... Like tears in the rain ;_;
>>52049262
>raid0
You deserved that and you know it.
>>52049425
Yeah ( ._.)
So my ssd on my computer has been running out of space. Do for Christmas I got a 240 gb ssd. I need to transfer the entire contents of my current ssd to the new one and then wipe the old one so I ccan use it for more games. How do I do this? Will I need expensive software?
>>52049877
Did you get a "desktop version" of the ssd? These kits usually have an oem version of Acronis True Image which you could use to copy all the data from a smaller ssd/hdd to the new ssd.
So we need a new way to build large disk pools since nonrecoverable read errors in pools > 12 TB.
Wat do?
>>52050157
ZFS.
I got this boring NAS for my hoarding.
>>52048446
Well, I have a 12TB 6x3TB mdadm RAID6 in my home server. I'm a bit worried because my drives are WD Reds, but when I made the purchase they looked much better in the data Backblaze had back thenthan they do now. I can afford to lose 2 drives and their failure rate isn't through the roof, so I should be fine I guess.
Just found out about smartd too and installed it.
Don't have any serious "offsite" or at least offline backup yet, except 2 drives (2TB and 1.5TB) that store some of my shit offline. I'll potentially need to have a look at more proper backup.
Also I'm very happy with mdadm and it's performance, it's been working great and has shown itself to be very reliable to me. The fact that you can mount your RAID in like 10s even from live Linux still feels like fucking magic. It's quite fast for sequential access too.
>>52050105
It's a Samsung 850 evo. After looking at the instructions it appears all I need is a usb to sata bracket so that the software that came with it can clone my current drive.
Now it's been 2 years since I built this thing so I'm a bit rusty. If I plug the cloned drive into the same sata port that the current drive is plugged into will the motherboard know to boot from that drive automatically or will I need to go into BIOS and set the boot drive again?
>>52049262
>RAID0
>movies
Fucking why? You'd literally be better off just using the 2 drives independently.
>>52048446
gonna build a NAS this spring along the following lines...
> Lian-Li PC-Q25 mITX case
> Supermicro X10SDV-4C-TLN4F Xeon-D mobo, with 1x32GB or 2x32GB ECC to start with
> 512GB Samsung 950 Pro m.2 SSD card for read cache
> 4+2 x 6TB HGST HDDs in RAIDZ2 in ZFS on Linux
> Distro not picked yet, but probably CentOS or SL.
> Gonna chuck a ebay special 1x 10GBASE-T intel NIC into my 2 main workstations with just crossover patches until 10GbE switches fall a bit further in price
My current setup is Synology, which I love for ease of use and flexibility, but you can't get a comparable higher-end setup (10GbE plus ECC support) from them for under several thousand dollars.
>>52050760
It should auto boot if youre cloning it
So I got a weird problem, every time I try to boot my PC while the SSD is connected, my PC refuses to go past the "press del to enter BIOS settings" screen. It won't take any prompts from the keyboard either. If I unplug the SSD, my PC boots normally with my main 1TB HDD. It also doesn't help to unplug the HDD and just use the SSD, the same thing happens, it gets stuck at the BIOS screen. What gives? The SSD is not broken. My mobo is z97-p.
>>52050157
RAIDZ3
>>52050862
Thanks. Now I'm just hoping I can find one of those usb adapters at best buy tomorrow so I don't need one shipped.
Do you *need* ECC for ZFS? My current box has 32GB of RAM, that should be enough for 20TB right? What about ZIL and L2ARC?
>>52051156
scrub of death is a FreeNAS forum meme, so you probably don't really need it, but then again, you never really know for sure, and ECC isn't THAT expensive.
ZIL is just for faster sync writes, so unless you're running a database over ZFS, don't bother.
L2ARC is fine, but general guidelines are to max system RAM for normal sector cache before throwing in L2ARC SSDs.
>>52048446
I have a noobish question- what's the breakdown of your data? like what % is music/movies/tv/gaymes/programs? is there any one specific hobby/interest which takes up a significant portion of your space? I don't code so i have no idea how much space compilers and such take...
How do you backup a 12 TB NAS? External hard drives? Another 12 TB NAS?
>>52048500
zero protection
raid 1: one extra copy
raid 5: don't have more than 5tb or else nrre
raid 6: you will get raided by women wanting 6 (sex)
>>52051305
Media files (music, videos) are about 80% of the data on my nas, but games (eg gta 5 60+ gb) are lately catching up.
>>52051318
you make a mirror on another machine, do a local network sync, then move it off-site and do syncs over VPN.
>>52048918
raid inandof itself is not a backup because any backup should be immune to any single point of failure
i.e. your house burns down == you are fucked
ideally your backup should include but not limited to:
multiple copies & various media
raid
offsite
optional:
ups
airgapped
rackmount
underground secret lair
>>52051354
do you actually have your Steam folder on a NAS?
A single HDD can saturate 1Gb ethernet nowadays, and almost nobody runs 10GbE at home for storage yet.
>>52051301
Sure it's a meme
>>52051407
general hierarchy of backup solutions is
> single external backup drive on-site
> two external backup drives, rotating leaving one then the other off-site
> RAIDed NAS on-site, external backup dirve off-site
> local and remote NASes, scheduled syncs
you can do shit like cloud storage too if you don't mind every kike and government agency on the planet having access to your data too.
>>52051351
>raid 5: don't have more than 5tb or else nrre
That's total bullshit. I'm running several raid5 with 4 3tb disks fine for more than 4 years now.
>>52051426
Yes, I don't have a hdd in my desktop and 1gb ethernet is fast enough for games.
>>52051519
>> single external backup drive on-site
oh god really?
>> two external backup drives, rotating leaving one then the other off-site
i wouldn't want to cart a box of hdds around. even moving tapes around makes me nervous.
>> RAIDed NAS on-site, external backup dirve off-site
my minimum acceptable level but i'm way too lazy to manage backups manually
>> local and remote NASes, scheduled syncs
living the dream. very simple ssh rsync.
>cloud storage
i thought we were talking about backups?
top kek.
>>52051526
>I'm running several raid5
yeah i have one with this exact setup
but i have never had to repair or rebuild an array
and you have to put up with people telling you how unsafe it is because of sensationalist articles they read online.
are seagate hdds and kingston ssds a meme?
I used to have a dedicated file server but I found I rarely used it. then I tried using a pi2 and it was so fucking slow it made for a bad time. Migrated everything over to my gaming build.
>4x1TB drives in RAID 10 using a dedicated RAID controller.
>Seagate 2TB HDD
>Toshiba 4TB HDD in a ventilated external enclosure for backups
I know I'm small time but it is more than efficient for me. Fucking HDDs are getting faster and faster though I'll say that. My RAID 10 setup only does 225mB/s sustained writes while my stand alone 2TB will do up to 185 and even the 4TB does 190+ over USB 3.0.
>Kind of want to buy 4x4TB Toshiba drives for RAID 10.
If anyone is unaware, Toshiba 4TB are the only 4TB drives on the market to have 129mB cache at 4TB. Everyone else only has 64. The drive is only $120 too.
>>52051638
I personally own several Seagates of varying ages and all with heavy use. 1-2TB models are rock solid and just as reliable as WD or HGST. As far as anything above 2TB, I wouldn't.
>>52051471
scrub of death involves random bit flips that trigger a SHA256 collision.
yeah, it's theoretically possible, but it's more likely that every member of your extended family will be killed in separate lightning strikes, so it's a question of how sane your priorities are.
likewise, you can still get uncorrectable multi-bit RAM errors with ECC if you're not willing to discount MTBFs of trillions of years or whatever.
>>52050157
It's called use not shit RAID cards and set up patrol reads so disk errors get found before they're a problem.
Somebody here running 20+ tb raidz3 arrays and deduplication? How much ram do you have? Write speeds?
I buy used HDDs from swap meets, as well as people I know who buy old storage units.
I try to stay away from IDE connectors, until I buy/find a IDE/SATA Adapter.
>>52048446
>tfw 2x 256GB SSD in raid so effectively only 256GB of storage
I don't know that feel OP
Why do people pay for SSDs?
>Writing speeds fast
>Mass storage fast, but deteriorates.
>Instant laods.
>Expensive as hell.
Why not wait?
>>52054920
Not everyone is poor, and consumer drives have been written to for over a dozen terabytes without failure.
>>52055013
Consumer drives as in HDD or SDD you mean, or both?
>>52054228
in-band block-based dedup is overhyped.
it's basically only close to worth it when you're hosting a lot of VM images on the same FS.
It also doesn't address listing bloat where you have like 5 different versions of the same reaction pic saved in different directories and you care more about killing duplicate listings than saving a MB of space or whatever.
Nice, haven't seen a hoarding thread in a while!
I consider myself a data hoarder. I download lots of images and videos on the internet and via torrenting.
My main storage drive is 3TB and important shit gets backed up with the 3-2-1 method, which means having atleast 3 total copies of your data, 2 of which are local but on different devices, and at least 1 copy offsite.
I have 2 main folders on my PC for dumping shit to. "firefox downloads" and "torrents". Once a year i do a really extensive spring cleaning, where I organize my downloads into folders elswehere (car pictures in one folder, NSFW webms in another).
I recently got fiber internet so i'll be looking into getting a NAS soon, but right now everything i have is a router with a USB port on it, meant for attaching harddrive so it becomes networked. Is that solution any good for a basic user? Surely it must have some password so family members can't snoop through my stuff
>>52055045
SSD
i plan to buy 4tb or bigger hdd for my desktop, any faulty models i should be aware of ? WD, seagate, hgst ?
I just cleaned out my hard drives recently. Got rid of all the old ones which were noisy and low capacity. Currently got 4.5TB of storage right now (about 1.2TB free) but I'm getting another 4TB drive for hoarding purposes of films and etc, organisation and for backups of the important stuff on my current drives.
Am I going to become a digital hoarder?
>>52055634
You have 3.3 terabytes of crap, you already are.
>>52048446
I got a FreeBSD home server with 4x3TB disks on ZFS 2x2 mirror which is basically raid10. I use LZO compression on all the datasets.
