Data Hoarding
Images are sometimes not shown due to bandwidth/network limitations. Refreshing the page usually helps.
You are currently reading a thread in /g/ - Technology
You are currently reading a thread in /g/ - Technology
/g/ What do you archive? Games, Music, Programs, etc...?
Why do you archive? & Ease of having it there when you need it? Feeling superior to others by having more 1s & 0s?
How do you archive? Any guidelines you follow?
& A follow up question, would you be interested in a website that that specialized in categorization of all that data?
Actually have tons of backups. Sad thing is, I don't label them.
Bibliotheca Aninomia
irc.rizon.net
#bibanon
anyone noticed how WD removed WD blues and merged it with WD greens aka it's the same as previous greens?
>>52358701
I archive stuff that shouldn't have ended up on the internet. (And likely no longer is)
>>52358701
Misc internet videos I'd hate to see gone if they are taken down. Tons of 4chan images, few movies and tv shows. Kept on a 1tb drive that recently filled up so I added a new 2tb drive for more room. Also got a 1tb external for backup, Decently organized but could use A LOT more work.
What kind of drives should I be buying?
>>52358701
I have hentai - in most all of its forms.
My home server has a 3TB drive for TV Shows, nearly 2TB full. A 3TB for Movies which I have 825 on there and it is just over 1 TB full. Also has a 1TB drive for pics, music, cell phone backups and random stuff. Also have 2X 480GB SSD's in RAID 0 as a scratch disk for writing torrents to.
I love these threads. I want to get into data hoarding with my favorite shows and books, but don't have the money to get a huge size hard drive yet (car payment is taking up my cash)
>>52358793
how did you get internet in there jared?
used to hoard just about everything due to OCD but stopped when I moved in with my bf
>>52358821
not seagate
I'm in the middle of sorting through 8TB of unsorted shit saved from 4chan, games, movies, personal projects, family photos, documents etc. etc.
I really, really wish there was some sort of template for prim and proper file organising. I'm terrible at it.
>>52358957
>bf
>pretending to be a girl on the internet
A lot of pics and webms from 4chan. Does anyone know a good program to sort and tag webms and pics?
>>52358771
W-whats this?
i archive chinese cartoons.
I used to archive and drink heavily because I was alone
There already is a site for categorization. aniDB. I parse my files when they are downloaded and auto rename them.
>>52359143
Would be in interested in this as well. Any other sort of helpful programs that assist sorting and organizing too.
>>52358701
i have approx 5TB of HD (bluray quality) porn.
>>52358886
>825 movies
>1 TB
how's that 640×480 treating you? can you tell which pixels are what?
>>52359196
Compression?
>>52359207
Averaging 1.2 GB per movie there, which is yify tier. Never again anon, never again.
>>52358701
Mostly anime. I have about a terabyte of roms, a few hundred gigs of music, lots of chinese cartoon porn comics, huge amounts of books be it novels or academic, and other stuff.
I'd like to make a dedicated NAS with some reduncancy as an extra backup. Those two 4TB drives are anime and games while one 2TB extrenal serves as an OS backup with some other stuff.
>>52359341
do it, i couldn't handle that mess
>>52359341
Do you have anything related to Internet archiving?
(Tons of sorted images or 4chan threads or OC compilations)?
>>52359389
I know where everything is, so it's not that much of a problem.
>>52359396
Not really. Archiving every post from 4chan would be nice, I'd also like to archive something like mangatraders or our goddess. I'm thinking of going risky with three 8TB Seagate drives in RAIDz, but I think going with six 4TB WD Reds on RAIDz2 would be a better idea.
>>52359196
Lots of old movies and kids movies.
>>52359430
Well, if you're interested, we do just that.
Come and join us on the IRC.
>>52358771
We are interested in saving any old or dying parts of the internet.
>>52359158
IRC... Probably some 'library' on it where you can get books or something
>>52359476
I just Googled it.
http://www.archiveteam.org/index.php?title=Bibliotheca_Anonoma
"TheBibliotheca Anonomais an internet research library focusing on archiving and organizing the history, folklife, and memes of the internet for the benefit of future generations. It is based around a Gollum wiki with a Git backend, for high portability and easy archival. The organization often contributes to the work of the Archive Team."
i hoard music
I'm thinking about building an 8 drive NAS. How bad would it be to buy 5TB WD externals and pull out the internal drives use the widdle program to set the park timer to 300 and then use those in an array?
Currently in the process of moving all my stay over to a 32TB RAID array. Mostly TV shows, ebooks, music, movies and the like. Some interesting things like the geocities archive. Mostly junk I'll never touch again. You know how the hoarding game is.
>>52358795
Do you by any chance have a Let's Play of Zork that was taken down on youtube? Either the profile went down or the video was deleted. Either way I used it all the time for white noise when studying because I like the typing and the guy didn't talk at all and it's been killing me that I can't find it because the sound is so ingrained in my memory but no other "typing noise" videos compare.
I'm about to have a new home built and my man cave is going in the barn loft. Does anybody have suggestions for storing and accessing movies, games, music, and books? I plan to have a purely digital library accessible from any where in the house via wifi on tablet, phone, desktop, etc.
I know this is all available on the web, but I value your input. Thanks /g/ents.
I hoard shitload of movies and music primarly.
Also lots of porn, obviously.
Currently I'm downloading all the YIFY 1080p releases, about 2000 movies downloaded.
Music I'm downloading because I have a dream that whenever my friends asks for me to play a song, I have it.
Check out reddit.com/r/datahoarder it's pretty good community actually.
I had less than 3GB free until last night when I deleted a few old torrents. I just can't delete anything as I might need that file at some point, but in reality I know that most of whats on here is crap.
>>52361663
Get another hard drive, noob.
>>52361708
Its an older laptop and I don't really want to put any cash into it. Besides, the problem is my hoarding and not lack of space for my collections.
>>52361730
Data hoarding is okay, because it takes up little room and prevents real hoarding
>>52361762
Thats my reasoning too. I also take photos of things IRL that I'm throwing out, this allows me to have a record of the stuff that I might have needed.
>>52361592
>YIFY
get out
>>52359180
where did you get all of that?
i can only find shitty 480p torrents
>>52361821
Decent quality and small filesize.
I don't understand what's wrong with them.
What I should download?
>tfw 2.7 TB of ebooks
>pdf, epub, mobi, chm, html, txt, docx, odt
>over 3 million different ebooks including the complete kindle-store, gentoomen library, tor library etc
>no I've not read every book
>no I will not finish 1% of it in my lifetime
>>52361147
NAS
>Hoarding
If I don't think I'll be using the data I'll get rid of it. I only keep things I think will be hard to find, like the untouched bluray rips of Planet Earth: Special Edition or hard to find PS2 ISOs.
Most of my data is on my 4TB storage drive on my home server. The more important data, which at the moment are my what.cd downloads and family minecraft server, are on my 1TB RAID5, which also has incremental backups to a 1.4TB partition on my server's boot drive.
I also setup a directory to my RAID5 via samba so I can put really important stuff on there if I need to, but so far I haven't done that.
The only thing I really like to "horde", as in download and forget, are cute anime pictures.
Pic related.
1.6tb of movies/anime ( 95% is blu-ray/DVD rips that i encoded myself)
About 400gb of misc stuff.
4x1tb hdds in raid 10 (2tb useable)
Stand alone 2tb disk
Both backed up to an external 4tb i put inside an enclosure.
Is there such thing as a picture re-namer? Like a program that will rename all pics in a folder by numerical order? Wallpapers etc i usually dont name and having several "defaultmaxres.jpeg" pics gets to be a hindrance.
>>52358701
I've owned alot of seagates but they are nothing compeared to fujitsu I STILL have my IDE HDD and its still goes LOL
but i hoard any data be it games , porn , family photos , bissness shit, homemade videos and movies that i paied for on itunes.
I have a mother load of stuff that very few devices can hold.
It will problaby be 15 years in NZ before any Large SSD because cheap enough to buy.
60c USD = 1 NZD
Life is suffering anon.
Programming books and video tutorials, over tb of it. Luckily all named correctly so i easily find what i want to learn on specific days
>>52362047
>4x1tb hdds in raid 10 (2tb useable)
Why would you do that? RAID5 would offer better redundancy and more space at the cost of write speeds, which don't matter much on a bulk storage device.
>>52362047
Are you retarded? There are dozens of programs for renaming pictures or pretty much any file.
>>52362011
>untouched bluray rips of Planet Earth: Special Edition
jelly
>>52362083
It was on rutracker and took me months to download since it had one shitty seed. If I had some place to upload all of it I would, but it's like 150GB.
>>52362092
I once wanted to dl Planet earth and Life in bluray but i didn't have the space for it, now i'm just lazy to look for it again and wait for days/months to dl, i'd rather buy it but i can't find it in my country
>>52362069
The array is also used as my work space. I get 250mb/s dedicated write speed on the array so it helps when writing heavily. I.e. large pictures im editing or 50+gb blu-ray rips. Im using a dedicated controller too because ssds are using all the sata ports in my motherboard. The controller only does raid 0, 1, 10. I tried raid 5 in the past using the motherboard raid chip and it was Just too slow.
>>52362070
No need to be mean. I've never had need of one until now. What are a few you'd recommend?
>>52362126
>>52362126
You're right, sorry anon. Personally I use Bulk Rename Utility both for images and general files, easy and simple to use. There are others like Rename Master and Ant Renamer that can use wildcards, but ive never had the need for them.
Im sure image editors like IrfanView, XnView and FastStone have renaming tools designed for images with more powerful options.
>>52361857
remuxes
>>52359389
please kill yourself
>>52359430
Blue dropped sometime around christmas to share about 3/4 of mangatraders db
Should still be on the archives
>>52360976
unfortunately not.
>>52358821
I choose WD Black most times. They are more expensive than other WDs, but have 5 years warranty. Meaning chances are better they are long lasting.
>4x4TB WD Reds
>1x 3TB external
>2x3TB arriving tomorrow
Mainly porn, TV shows, movies and a fuckload of NEF files.
I only buy drives in pairs now for mirrors.
>>52358789
when?
I've got 3 external hard drives (a total of 12tb) full of movies, tv shows, documentaries and music. I'll need a new one soon but would rather just replace all of them with 1 large desktop hard drive, just don't know what to get. Any ideas?
I've read that the 2.5" reds were shit compared to the 3.5" ones, Anyone used them?
>>52358789
Pretty sure the removed the greens because they were shit and kept the blues and the now cheapest drives
>>52358701
Mostly music, images, deleted/rare videos and anime.
Porn lots and lots of porn.
>>52359143
>>52359168
For you and anyone else that needs to sort their shit.
https://github.com/hydrusnetwork/hydrus
http://hydrusnetwork.github.io/hydrus/
Hydrus does
>images - bmp, gif, png and jpg
>audio - mp3, ogg and flac
>video - webm, flv, mp4 and wmv
>misc - swf and pdf
>can also grab tags from your weebshit sites(boorus, pixiv, ect) tumblr, deviant art, newground
>or make tags of your own
And a bunch of other shit.
>>52361858
I need this anon
>>52365948
The tags are in a database right? This doesn't actually modify the files, right? [Just being a tiny little bit paranoid here]
>>52366160
It basicaly make a copy of your images into the hydrus folder or network(be sure to have the extra hard drive space). You can then either move or delete the originals and nothing will change in hydrus. It discards their filenames once its in hydrus and uses the image's hash. So if you want your original images back, if you deleted them, you can just drag and drop them back out of hydrus and nothing much will have changed.
You are going to want to play around with it until you get a better understanding. Read the help section in the 2nd link, it has a nice tutorial. Just play around with just a few of your images until you got the hang of it.
Mostly hoard scene content; tv shows and movies.
>>52358701
>/g/ What do you archive? Games, Music, Programs, etc...?
Mostly images and videos
>Why do you archive? & Ease of having it there when you need it? Feeling superior to others by having more 1s & 0s?
if I find something I like, it gets saved
>How do you archive? Any guidelines you follow?
I'm using the 3-2-1 strategy (3 total backups, 2 of which are on different devices and 1 offsite). Every hard drive is full disk encrypted aswell.
I'm using external hard drives but high capacity flash drives are dropping in price so I might just snatch one and put on my keychain for ultra mobility
>& A follow up question, would you be interested in a website that that specialized in categorization of all that data?
Sure, hit me up senpai
I'm also in the middle of my yearly spring cleaning and organization (because i download everything into one massive folder)
>1 4TB WD Blue Hybrid
>1 4TB Seagate Hybrid
>2 4TB Toshibas
>1 5TB Toshiba
>1 6TB WD Red Pros
>2 4TB WD Reds
>1 4TB Hitachi NAS
>two-port mini-SAS HD RAID card and SFF-8644-to-SATA3 cables on the way
What do?
>>52366423
>I'm using the 3-2-1 strategy
Damn. That's very expensive but I get you after losing about 3TB of anime. Good thing I had one drive with backup.
Where do you keep your offsite backup?
>>52366429
I would ZFS.
>>52366447
at my parents house, encrypted of course
1.34TB of music (70% lossless)
850GB of movies/animu etc (includes ~110GB of porn)
650GB of recorded stream/video footage (from myself as well as others)
100GB of Stepmania song packs
Currently saving up for a 800GB-1TB NVMe SSD, and possibly also a 1TB SATA SSD as part of a new build.
>Why do you archive?
Basically the only big reason I can think of is that I've lost track of how many times I've been wanting to find something, only to realize that it's no longer available online - since as far as I've noticed, the idea of things "staying on the internet forever" only holds true if plenty of people are actually interested in it. Also, it's kinda comforting to have pretty much everything I want within my direct reach...
>>52361592
I did this before. It's honestly not worth it, anon. Like 80% of those movies are utter shit and I only got through watching like ~600 of them.
>>52358701
I have a network hard drive. How can I combine multiple hard drives to appear as one on the share?
>>52366468
Which ones should I ditch? They're all under a year old
>>52367216
run a smart disk check and chuck the worst one
>>52367869
All of them are at 100% or Good. The worst I see is a 4TB Toshiba drive that has G-Loading error value of 5.
This probably belongs in sqt but how do i get a tiny 256gb c drive to write to another drive before it runs out of space? I have 2 3tb wdgreens, a black and a blue and a tiny ssd c drivd and this is annoying as fuck
36TB divided into two volumes in a RAID5 DAS. The PM is Addonics and the case is homemade from a toolbox and hold both sets. I build my friend a similar setup and we act as each others off site backup. I use gui front end to rsync keep them the synced.
I store mostly anime and japanese related stuff on them, but there's also US stuff and whatever other animation gems I can find from from other countries.
I had a neat idea. How about having a large public database of various hashes to what a file is?
>>52368218
>change 1 bit on a file
>old hash wont match
there isnt enough storage space on earth to support your idea
>>52368172
>RAID5
>/g/ What do you archive? Games, Music, Programs, etc...?
I collect and trade live music recordings, these are generally recordings done by fans in the audience or they're full shows broadcast on TV or radio, I don't collect or trade anything that's been officially released by bands. As of right now I have approx 1.5k full live recordings in my collection all post processed by hand all in FLAC.
>Why do you archive? & Ease of having it there when you need it? Feeling superior to others by having more 1s & 0s?
I love collecting/trading live recordings. It's something thats been going on for decades since Grateful Dead let fans tape their shows but has now moved into being mainly torrent/FTP based downloading and sharing. Back in the day trading of live recordings was much easier given that it was physical media based so all you needed was a text list of the tapes or discs you had in your collection but now it's moved to online torrenting it becomes much more difficult.
>How do you archive? Any guidelines you follow?
It all goes on a large 2TB hard drive and I also upload a lot of the better shows to a blog to keep them around. The areas of music I'm into are fairly niche so if they aren't downloaded when the first appear they can disappear forever.
The file management side of things is an absolute nightmare to deal with however. The problems with the files begin with the initial distribution, while there are many trackers out there for live recordings the largest by far is DimeADozen and it's generally where all new recordings first surface. The main problem is there is no file name or metadata standards on DAD whatsoever, I regularly download sets which are folders named "Artists Name - Show Date" and the file names are 01.flac, 02.flac etc with no metadata. This is useless when it comes to having a large collection as once something is older than a few weeks it can become impossible to find them again or even know what they are sometimes.
>>52368382
So to counteract this I rename and retag everything to the Etree standards to provide a standardised formatting in my folders.
For most music tagging of metadata is easy since there's standardised databases for info (like Discogs or MuzicBrainz). For live recordings there's only the one database out there (db.etree.org) but it's really only useful for the more well collected artists such as Grateful Dead, Bob Dylan, U2 etc. A lot of the artists I collect (the areas of punk, hardcore, noise, experimental etc) usually don't have anything on there. So I run any files I download through MP3Tag with a custom set of paramaters and try to keep things the same for every fileset. After that the file sets are run through Trader's Little Helper to create new MD5 and FFP files for future verification.
Of course the major flaw in renaming/tagging files is that I can no longer seed them on any sites and have changed the checksum variables. While to some this is a problem I consider the trade off of having an actual useable/readable collection worth it against not being able to seed back. I also record a lot of shows myself so ratio and the like is never a problem.
Once the files are all renamed and retagged properly then comes the problem of cataloguing. While I do keep track of the library in Foobar I do also like having a list for trading with people. Etree was the place I had the list for years but I grew wary recently that the site may close (it's very inactive) so moved my cataloguing to a peice of software called PHPRecDB which is used by a few others to create lists of live recordings. Again, this is all a manual process where I chave to fill in approx 10 fields to create each individual entry. Yes, I've done this over 1000 times now.
I don't normally care about data hoarding but I'm borderline autistic when it comes to dealing with these filesets.
>>52368251
It doesn't make sense for all files, but a lot of things have a canonical bit representation like OS images and installers/packages or audio/video releases
Porn. 6TB of it.
>>52368423
that's only useful for confirming if files downloaded error free.
Do I have to format my hard disks if I make them a logical volume? I don't want to lose my data, but I want my hard disks to appear as one.
>>52368289
With offsite backup.
I plan making a real server as my next storage solution, but I don't have budget right now.
After losing some data:
Prio 1: your photos and your works. Don't delete anything you made. Legal documents.
P2: Stuff you downloaded or organized. Eg images from 4chan, and the like.
P3: anything else
2 copies of everything, 3 of important stuff. HDDs will fail. Keep a list of what you have on each HDD, it can help if a HDD fails.
Welp, most of my archived stuff are chinese cartoons.
If it's worth it: The best thing to stumble upon is "treasure", like my old download directory from older system drives, programs/pictures/threads I saved, and so on.
The most useful backup was a clean system install with all essential software I use. I have it on a pendrive that boots into windows installer, and another one that boots into Hiren's.
I never replayed most of my archived games. I never rewatched most of my archived anime. The older anime I kept from the 90s and the like is so low resolution, it's almost unwatchable. Sometimes it's just there to remind me, and I try to dl them again in a higher resolution if I want to watch them again. I still think the archive is useful, at least to remind me that these things existed. Do you remember what you liked 10 years ago? Even if you do, it will never pop into your mind if you don't stumble upon it again.
>>52368539
>Keep a list of what you have on each HDD
This saved me when I was in the middle of backing up a drive. The drive died copying stuff, but the list allowed me to download most of the anime and music.
>>52359341
>>52359389
>Movies
>Music
>Some pr0n
>Pictures of your mom (Those are the big files)
>Random downloads i refuuse to delete
>Too many gayems
>Random ISO's of LUnix distros/Windows
Got some decently old drives
I wish I'd never gotten into the PS3 ISO scene. Even just by keeping the games I actually like I'm at several TB's of games/DLC's. There's people I know out there that have archives of every single scene game from every region and it's jsut massive.
>>52359019
He probably is a faggot
This is probably a good thread to ask, what is a good 2-3TB SATA III 7200rpm HDD that is as quiet as possible and not insanely expensive?
I have all of my video stored on an older 1.5TB WD that's near the end of its life and the droning noise it's starting to make is driving me fucking insane.
>>52368706
HITACHI
I
T
A
C
H
I
>>52368706
Get lower rpm for quiet. It's not like there are many types, check what's available and choose one. Most attitudes on HDD quality are based on anecdotal evidence.
>>52358701
I think there is a difference between hoarding and collecting. I archive movies, music, tv shows and books. Once I have them downloaded I watch them and keep it on my harddrive incase friends want to see it too or I want to rewatch them sometime again
Everything.
Have like 4 full 2Tb external USB drives with mostly software and games. Then I have a 4Tb synology cloudstation with photos on it. Also two 2Tb WD black hdds with everyday stuff on them. Defrag, Defrag, Defrag, that's my recommendation
>>52368706
Just pick the cheapest but ignore greens and Seagate. I just bought 2x Toshiba 3tb for £65 each.
>>52361858
Where can I get this?
>tfw you have a RAID'd NAS but no real backups
>>52372645
yo fucked mate
i stopped archiving shit when i got older. i'm tired of hdds breaking, i'll wait until ssds are cheaper before i start my media library again.
>>52358821
toshiba
>>52361858
i hope you give shit out in person to people
>12TB NAS for films and shit
>3 TB in computer at the moment
Don't really hoard, just have quite a few blu-ray rips and download all my TV shows in 1080p 5.1 if I can.
>>52361858
The installgentoo FTP has over 2TB of books, btw
>>52372645
Next thing is you're going to tell me it's a 40TB array with RAID5 on a Windows server.
>>52359168
>>52366423
>>52368382
Kodi is the best organizer for movies, tv shows, music, anime, pictures and porn. It automatically matches official movie covers, descriptions and IMDB ratings to all your movies and you can browse through them easily like netflix. data hoarders
should know about Kodi.
>>52375007
>>52375027
So like Plex but it also supports porn? Neat.
>>52375007
I didn't like kodi. Plex needed less fucking around. It just werks so to say.
>>52374025
How would you back that up anyway? Another identical raid5?
>>52358701
Nothing, I have hordes of literal retards with too much time/HDD space seeding media torrents and/or uploading it to direct download servers.
Only thing I save is music albums that are hard to come by.
Are there any off the shelf NAS options that come with a physical unit as well as cloud storage included, either a 1:1 ration of the storage on the physical device, or less just to serve as a backup?
I think the big question here is how do you store your data?
Magnetic strip?
SSD?
HDD?
>>52369553
> defrag
> lol he doesn't ext4
