I have a somewhat large (~5TB) pornography collection.
I'd like to make a database with all the proper tags for easy location of content. I have experience with MySQL, but would PostgreSQL would be better for something like this?
Wow.
>taking porn this seriously
I heard of something called Hydrus, dunno if that works with video though. But it's worth looking into I think.
Considering this is something you'll mainly be using yourself there's no reason SQLite wouldn't do.
>>53526275
Seems like it only works for images. Looks good, though.
>>53526255
What?
This thread makes my 1TB collection seem less weird. Thanks OP.
Back to the topic at hand, if it's primarily images (probably not, I know) you could look at running you own *booru:
https://github.com/r888888888/danbooru
https://github.com/moebooru/moebooru
They're mainly based around Postgres
I've never really found a solution that works out for my own personal needs. Good luck.
>>53526382
It's mostly videos and doujinshi/H-manga.
>>53526082
>>53526382
>having a porn "collection"
>>53526082
>5TB
Lemme guess, 90% of your collection is weaboo series? I have 250Gb of porn on this computer, and an extra 55Gb of porn that I could may or not get in trouble for on an external. Unless you just mindlessly download a bunch of random series you can't have 5tbs of porn... unless you started your collection like 10 years ago
>>53526082
use mongoDB and a local host
>>53526421
Not 10 years ago, more like 7 years ago.
I used to download everything i could get my hands on, but I have recently vetted every entry and trimmed off a good 2TB.
>>53526343
>What?
Dude, you're storing, indexing, cataloging your porn.
Just do like normal people and just go to Pornhub or something, jesus christ
>>53526343
He said your an idiot for giving a shit about porn. What the fuck do you mean "What".
>>53526420
>not having a porn collection for the inevitable collapse of civilization and ascending to become the king of unmentionables by subsequently trading your illicit wares.
>>53526456
>Dude, you're storing, indexing, cataloging your porn.
If it was literature or music it wouldn't be odd in the slightest, why is porn any different?
>I have experience with MySQL, but would PostgreSQL would be better for something like this?
Other than the general reasons why mysql is better than postgresql, it should make absolutely no difference.
Apache Spark will work better as you expand into exabyte storage clouds.
>>53526476
>If it was literature or music it wouldn't be odd in the slightest
It still would, because there is half a gorillean software applications that do exactly this already.
>>53526485
>mysql is better than postgresql
keke
>>53526476
>why is porn any different?
This is why you'll never leave the basement.
>>53526475
In the case of the apocalypse, why settle for mere porn when I can trade stuff in order to fuck your wife's daughter.
>>53526540
I own an apartment, Anon.
>>53526559
>own
>apartment
Kek
>>53526559
>I own
You mean your bank owns.
>>53526559
That's even sadder, desu. Why tfw no gf?
>inb4 3dpd and i want to fap to my waifu
>>53526311
SQLite is great tool, but for such huge collction you might conider MariaDB or PostgreSQL as it give you more flexibility.
I have only 3TiB of images and i use MariaDB, all metadata takes around 16GiB.
>>53526605
I was in a relationship up until 7 months ago.
>>53526653
And look at you now.... Asking a bunch of 12 year old strangers on the internet what's the best way to index and catalogue your porn
Isn't life cruel, anon?
>>53526082
Here's a simpler solution
>take the hard drive with all the porn
>Strap it to your fallen dreams and lack luster social skills
>Throw it off a cliff and try not to remember ass assassinator 3: director's cut
>>53526553
brb, researching human slave trade
>>53526681
This
>>53526678
>Isn't life cruel, anon?
>>53526421
your a pleb
i filled a 1tb drive with just scenes i want and nothing extra since autumn
good quality 1080p scenes go up to 4 GiB
you do the math
>>53526082
You already have a database of them, it's called the filesystem. Use extended attributes to put custom tags on them.
>>53526082
Use TagSpaces.
>>53526082
>Only 5TB
Faggot
Does Hydrus Network support video formats?
>>53526678
welp
>>53526553
Because trading that porn, or renting it as a service, will create the funds that you will trade for both his wife's daughter, a guard for your business, and possibly also for his wife when you pay a bunch of guys to kill him in order to break his wife's pussy.
>>53526873
https://github.com/hydrusnetwork/hydrus
http://hydrusnetwork.github.io/hydrus/help/index.html
>>53526421
LARSON!
In the near future no one will need porn anymore because you can just have VR sex with your full-body-tracked silicone doll
>>53527584
* unless you're poor
>>53526082
>Organized 5TB PostgreSQL porn database on personal computer
Does that not strike you as weird in the slightest bit?
>>53527584
>Implying that data won't need to be stored locally
>Implying said local data store wouldn't be classified as pornography
Ta~da, porn stash.
>>53527644
Why would it?
>>53527644
You're right, should be stored on a home server for distributed local access.
>>53526082
Delete it you degenerate.
>>53526431
kek
>>53527690
It is on a server already.
>>53526082
Holy shit. just use gelbooru. or danbooru. or moebooru. or shimmie.
why the fuck would you make one from scratch if you're not using it as an excuse to learn something new?
>>53528353
>Terabytes of 1080p video
>Gelbooru
lol
>>53526082
holy fucking shit you fat neckbeard this board is just fucking sad
>>53528368
>implying adding support for videos doesn't take 10 minutes tops
>>53526082
Ebin
Hmmm. What if you train a neural network to recognize pornstars and make it index them for you on a name basis. Maybe you can do the same for tits size by comparing them with the head size. Aside from the fact it's for porn, it's a really interesting problem since you're profiling people by physical properties( pulling type of porn performed would be tougher I guess)
>>53529201
Already been done.
>>53526082
When will they make Plex for porn?
>>53526082
>5tb of porn
The fuck?
>>53529554
yes, 5tb of fuck
>>53526082
both are fine, pick whatever you're comfortable with
>>53526456
fucking casual parasite
why is mongodb a bad idea for this
>>53529554
I'm not OP and I have 5TB
and I'm small-time as far as data hoarders go. it's pretty easy to accumulate over the course of a few years.
>>53530314
It isn't, but the dataset he is building is likely very well structured, which is where regular sql dbs shine
>>53530344
>dataset he is building is likely very well structured
debatable, depends how autistically he structured his directories/filenames
>>53527670
Because normal people have a partner and they would climax together.
>>53530375
>debatable
I disagree, the actual path to the file should be just another column.
>>53526082
Either will be fine. 5TB might seem like a lot but it's metadata we're talking about here. Assuming your average porn file is shit quality 100MB video, that's 50,000 files. Even with 10 tags per file that's 500,000 tag-file relationship records. That's hardly a big number for a modern relational DB.
>>53530341
Yeah but 5TB of nothing but porn? I data hoard like the best of them but I don't have any single collection over a few TB and that's mainly PS3 games which can hit 50gb each.
You can make your own internal website to view those contents. I thought about creating that but I had a job....
>>53526275
>>53526343
It supports video.
>>53526255
It's not about porn it's about the DB management.
Why not just use something like VideoDB (http://www.videodb.info/)? It has a plugin for a bunch of porn sites in the forums or you can just manually add everything yourself.
>>53530314
Because mongo is a bad idea for everything.
http://www.mongodb-is-web-scale.com/
>>53526082
Regardless of content, this is just a document storage problem which has been tackled satisfactorily many, many times.
I'd just do a NOSQL DB with file IDs, file paths and various named tags such as:
* Publisher
* Director
* Media type (image/video)
* media subtype (full movie/scene/clip)
* related files
* performer (multiple tags)
* fetishes (multiple tags)
* genre (a broader version of fetish)
* filesize
* duration
* resolution
* file path
* file name
* title
With some scripting, some of this can be autogenerated.
Make a script to autopopulate the machine readable fields and then make a web page with a button to load a random entry 80% of the time. If the entry hasn't been manually annotated then you have to fill out the fields before the file is loaded for you, or it deletes the file.
You'll rediscover a lot of old porn, delete some of it that you're no longer into or which is superseded by better quality stuff, gradually index your collection and accomplish the primary goal.
>>53530776
Best answer in the entire thread. Thank you.
>>53530776
>dicksize not one of the tags
what are you, a fag?
>>53526611
What kind of inefficient shit program have you written for yourself to store 16GB of metadata for a measly 3TB collection of files?
>>53526082
>GUYS LETS TALK ABOHT TECHNOLOGY
>/g/ "you're a gross disgusting individual, fuck you!
Good goy let's go back to talking about GPUs and how to unlock a stolen iPhone
>>53530874
Truth
Why the fuck is /g/ so moral all of a sudden?
This shit seems perfectly reasonable. Anon just wants to organize his fap material.
>>53530996
An influx of normies and redditfags
Doesn't Kodi have scrapers for porn? I'm pretty sure it used to in the past
>>53530996
I miss when /g/ was full of loli. There'd be a thread of people talking about motherboards and they'd be posting loli while they did it, just because.
it was a more innocent time
>>53530816
>anything using non-relational
>good
Nigga you dumb?
>>53531057
Doesn't matter, that solution can be applied to a relational too
>>53526082
Who the fuck has 5TB of porn?
>>53531044
RIP techloligy
>>53531075
lots of us.
How come you don't?
>>53526082
Sqlite works pretty well for stuff like that.
>>53529356
Source? It would be a good exercise to do it yourself anyways.
>>53531368
Don't rememebr now and cannot find it in my favourites, however someone has written a scientific paper abut it and made a proof of concept weef some weeb images.
All i can rememebr is that it fingerprinted each image in multidimensional space.
>>53530776
So I've been considering this as well. In addition to mongo, I was also considering piping file path/metadata to elasticsearch. That way I can get an idea of whattt kind of content I have with lucene. Also have a ruby script to extract portions of the video to make a collage thumbnails.
>>53531044
I still have some pics of those days
>>53531399
Is it this one?
http://infoscience.epfl.ch/record/103736/files/EPFL_TH3797.pdf&ved=0ahUKEwiI3d-768bLAhVGbB4KHTzXAkAQFggkMAM&usg=AFQjCNFjLr7BiKHKjMAEOR4ky18Iv0deFQ
well hello this is my first comment on 4chan. :D
>>53531687
welcome to 4chan, fuck off
>>53531548
Nope, this one is quite old, the paper i'm talkign about was like 2 years old.
>>53526082
You could ask Google's AI for help, I bet they would like it to get some experience.
Found it:
http://illustration2vec.net/
>>53531818
This is great, thanks!
>>53530826
>>dicksize not one of the tags
>what are you, a fag?
I'm genuinely contrite that I didn't consider this.
However, lesbian porn being one of the likely subject matters, it might make sense to make that a fetish tag rather than a tag of its own.
Then again, a PerformerAttributes set of tags makes sense.
* innie [T/F]
* cut [T/F]
* length
* cup size
* beefcurtain length
* anal-bleached [T/F]
* gape-radius
Or whatever you want.
>>53531073
>Doesn't matter, that solution can be applied to a relational too
It certainly can, the only reason I prefer NOSQL for this is because you have flexibility with tags.
Otherwise you have some big table of:
tagID | tagName
fileID | tagID | tagValue
Easier to just make custom tags on the fly. At least that's the world I work in.
>>53526082
Use a taggable filesystem and tag them in your filesystem.
>>53531818
wow this is awesome
thanks anon
>>53526421
Don't you implicate Larson into this!
I use Hydrus Network, even for videos.
It sucks that it generates thumbs based on the first frame though.
>>53531818
right now with a simple script its sorting all my pic and adding the tags to a meta library
5 TB is a little more serious than "somewhat large", friend
>>53531818
fucking great man
>>53530816
>>53531073
That was the worst answer in this thread you retard.
NOSQL would be terrible for this kind of setup, think about how many fields are relational when it comes to porn:
- a video can have multiple tags, tags can belong to many videos (M2M)
- tags could have multiple levels or belong in categories, e.g. hair color > brunette
- tags could have a description
- videos can have multiple performers, performers can belong to multiple videos (M2M)
- performers have a thumbnail image associated with them and a bio description
- videos may belong to a studio
- videos may belong to a set
Do you realize how much data would be repeated over and over if he used your retarded MongoloidDB shit here?
Stop parroting your retarded NOSQL and other meme nodejs shit everywhere and kill yourself already. If you don't understand how something as simple as SQL works, don't use some terribly inefficient alternative, just spend one day learning how it works instead.
>>53526082
>sql
>2016
>grep
Stallman already gave you the answer. You've had it all along
>>53532908
I'm sorry pajeet took your job, anon
But with this many anger issues you'll never make it past even the phone interview
>>53526678
>implying there are better people to ask these sort of questions from
>>53526082
ITT: OP asks 12 year olds how to catalogue his 5TB child porn collection.
>>53530385
>climaxing together
An actual myth. Women fake it to seem more romantic or some shit.
>>53526255
I do this faggot.
I don't care if i have a problem, it's good practice for a few IT skills and it satisfies my autism.
>>53526521
And there are zero that do this for porn.
What's your point faggot.
>>53527690
I've already done this, I have two virtual hosts serving porn via a lamp stack to my whole house at over 1gbps.
Lol get on my level fags
>>53530473
You can and it takes like an hour.
Google shimmie2
Fucking hell how much time do you people spend watching porn? Is this some sort of depraved hobby?
At least you'll have experience for a dba position with the rest of the weird fucks.
>>53534302
I don't watch any of it, i just connect it in case
Sorting it is for convince and archival purposes.
>>53534320
Collect*
>>53534302
>Is this some sort of depraved hobby?
Yes. I like collecting it. It's all backed up, too, I'm watching rsync move a pile of JAV right now.
>>53526082
Don't worry OP
I also have 5TB of porn
Completely unorganized and I hate it, but honestly I don't have the time/patience to even go through and delete what I don't really like.
People just be jelly.
>>53534625
>Completely unorganized
>>53534674
Well maybe not 'completely' but close to it, just a few big folders for eh...orientations
>>53526082
I officially call you the masturbator, the guy who masturbate himself faster than his shadow
UNRELATED to OP's illness but I have a huge video collection.
Is there any program that is like Foobar2000 but for videos? I'd love it
>>53534956
emby/kodi
>>53535081
aaaaaaaaaaaaaaaaaRGH I DONT MEAN TRASH LIKE ***MEDIA CENTER*** I NEED A POWER TOO TO ORGANIZE A HUGE LIBRARY WITH TAGS, WIDE SEARCH OPTIONS LIKE IN FOOBAR AND PLUG IN SUPPORTTTTTTTTTTTTTTT
Somewhat related, maybe?
Any of you who rip websites, what programs do you use? I've been using pixiv-dl and some custom stuff as well.
>>53535105
Firstly, stop acting like an autistic retard.
Secondly, Kodi is actually similar to foobar, it does half the work for you (categorizing by folder and giving you a decent view of it) but both of them can't do everything for you because they rely on scraping certain databases to get information about your media.
That last bit is the real problem for using Kodi for porn, unless all your porn is just mainstream DVDs, in which case it can be scraped decently from APIs which include adult content. If it's content which can't be reliably scraped, then you'd have to resort to manually organizing, hopefully with the assistance of some scripts or software to do it in bulk where applicable, e.g. bulk assigning actors to all the videos which have that actor, etc. There's some software out there designed to work with Kodi which can generate the nfo files which Kodi uses, such as Ember Movie Manager, MediaElch, etc.
>>53535330
I don't need the work to be done for me, I literally just need foobar but for films, not a fucking server or something.
>>53535374
Kodi doesn't have to be set up as a server, you know that right? You can just install it on your local PC, scan a local directory and then view your shit with Kodi (just like foobar).
>>53535421
I watch my films with MPC-HC+madVR (and Lav)
how can I make it so that Kodi defaults to this
>>53535442
bumpo
>>53526431
>using mongo anything
>>53535607
>>53535442
How can I set up Kodi so that next to the poster or below it
I could see the director, genre, year etc.
and then click on it
and it'd show matching results from my library
>>53535794
HOLY FUCK BALLS THIS PROGRAM IS FNATASTIC
>>53535804
IT FUCKING UNDERSTOOD TO SETUP AKI'S THREE FILMS AS A TRILOGY MY FUCKING SIDES
>>53535810
>>53535804
How can I rename the 'original title'? the film is actually first REC but the movie db scraper misnamed it and fetched wrong info
>tfw 4 tb minimum of chaturbate recordings from letting rtmpdump run all night
I want to get rid of most of it but the good models take up the most space
>>53531075
People who don't delete the bad/mediocre shit and keep everything.
>>53535839
Does it work well with Anime?
>>53526343
>Seems like it only works for images.
video - webm, flv, mp4 and wmv
these are the video formats it supports
so basically just not anime desu
>mfw I have a measly 250 gigabyte to store all of my things (it's not an ssd either)
>>53526082
>1. Put them on a PHP / MySQL server
>2. Get traffic
>3. Get money
>4. Profit!
>>53526082
I just use Kodi and NFO files. That's for videos though. Kodi has a fucking awful way to manage images and shit.
>>53526082
I feel you, I have roughly 3tb saved, haven't yet found a good solution, to lazy to do it myself.
am I the only person who has a wank and then discards whatever they were watching?
>>53539757
Everyone here has specific fetishes that they can't just find easily on pornhub.
>>53535839
Rescrape with manual title entry
>>53536470
Yes but bitch-ass faggot anime naming isn't supported. Either use XEM scraper or rename your shit
>>53535794
>season
Don't use tmdb for TV shows, fuck's sake!
>>53540998
>Rescrape with manual title entry
I'm sorry, but how? I had to rename the file in explorer (windows explorer) and rescan. How can I rename it through Kodi?
Also if I have TV-shows and films mixed up in ONE FOLDER, like folder for Mr Robot s2 and there are files, but in the upper folder you might have few films (named like scene names them usually)
how can I set up Kodi to scan this parent folder which has both tv and film through both - tvdb and imdb/tmdb?`
>>53541018
Which one?
>>53526082
>somewhat large
>>53527690
>>53534058
This is really the best advantage you can get outta this
>Fap on the main computer
>Go take a shit phone in hand
>Fap again
>On bed with a laptop
>You know what it is
>>53541031
>>Rescrape with manual title entry
>I'm sorry, but how? I had to rename the file in explorer (windows explorer) and rescan. How can I rename it through Kodi?
Press 'i', select refresh and then manually search from there
>how can I set up Kodi to scan this parent folder which has both tv and film through both - tvdb and imdb/tmdb?`
You can't. Either make separate folders for TV and movies or manually set content type for every single folder.
>Which one?
Use tvdb for TV content
>>53531818
>>53532792
How do I use this?
>>53526412
Ofcourse it is
>>53526082
>tfw wrote software to categorize own porn collection
>>53542946
post screen
>>53526559
>I "own" an appartment
>>53542946
Thinking about making my own aswell, care to share your approach?
