Hello /g/.
I do a ton of archiving/extractions. I'm talking 10k+ archiving of folders daily.
What would ultimately speed up this process? A server-like setup with a shitload of cores? I can't think of anything else that would help performance.
Storage speed will most likely be your bottleneck. I'd start investing in that first.
>>55324029
try another fs
>>55324029
Need more info.
Amount of files, amount of actual data, operating system, frequency of file changes, etc. Also, stackexchange
>>55324029
POWER8
Don't some filesystems like ZFS and BTRFS have build in compression you can enable?
>>55324336
I figured, but I can't move to SSD's because It's too expensive to invest in large amounts of space for that medium.
So 7600 RPM drives it is.
>>55324409
Don't think my FS has anything to do with it
>>55326329
>Amount of files
The amount of files in the archive? That varies. I'm archiving thousands of folders which may contain anywhere from 10~100 images though.
>amount of actual data
Probably 1.5TB+
7zip supports multi threading so having 10+ cores would really help. I can probably overclock up to 4ghz for extra speed too, but it may all be pointless if it's bottlenecked by HDD speeds.
>>55326959
yes, but it's block level compression with fixed blocks
zfs has online dedup as well, while btrfs dedup has to be run manually and is more efficient because of it
>>55324029
use linux with the xz kernel driver. performs better on modern processors than 7z on windows. ntfs is a clusterfuck and should be avoided when possible
RAID 10 your shit for speed and reliability. BTRFS can do this in software with low overhead and then can scrub for bit errors. What would you do now if a file became corrupted? you'd be fucked
