Used to have a program that let me download all the images in a thread at once, but after moot changed the url to have the subject of the thread they all broke.
Anybody got a working one?
In the mean time i'll dump some stuff.
Anybody?
>>3814561
Downthemall is a firefox addon that lets you do that.
very nice pictures op, keep dumping pls
>>3814578
Thanks!
works well.
I'll dump the rest of this folder [30 images]
pretty sure its all uncensored.
>>3814580
:)
I like this folder but wish there was more cum involved.
>>3814581
No problem OP, glad I could help.
And that's it, happy fapping fellas.
If OP is still here, DownloadAll works well for chrome.
You can easily write a script to do that. Has the added benefit that if they change the url you can just change one line in the script and it works again.
>>3814689
Here is a script to download images from a given thread link.
[code]
use WWW::Mechanize;
my $mech = WWW::Mechanize->new(agent_alias => 'Linux Mozilla', show_progress => 1);
$mech->get($ARGV[0]);
my $content = $mech->content();
$mech->get("http://" . $2, ":content_file" => $1) while ($content =~ m,<div class="fileText" id=".+?">File: <a title="(.+?)" href="//(.+?)",g);
$mech->get("http://" . $1, ":content_file" => $2) while ($content =~ m,<div class="fileText" id=".+?">File: <a href="//(.+?)" target="_blank">(.+?)</a>,g);
[/code]
this is written in perl so just download it for whatever system you are running and execute it with:
perl scriptname.pl threadurl
ychan . . . source forge . . . google thats all im going to say
>>3814578
Is there a chrome version?
>>3814700
Thanks bro works perfect.
>>3816564
4chan image downloader.
It's kind of annoying tho, because you have to choose the saving location for every single image.
>>3814561
>url to have the subject of the thread
you can also just manually edit the url to remove the subject, that part of the url is fluff
Well, I just finished a very fast, very small program written in C for all POSIX-compatible OSs that would download all images of all threads that you'd put in a link file. The HTTP stack was completely written by me, everything uses persistent memory, and it's multithreaded, so you don't even notice it is running. It supports SOCKS4, SOCKS4a and SOCKS5 (without GSSAPI authentication).
And know what? I won't give it to use unless you pay me.
Prove that I really own such a program?
[2014-10-07 19:30:27]: [0x7f2529cfb820|139797591996448|"/home/anon/collectors/4chan_results/h/3807816/1412670867551.jpg"]
[2014-10-07 19:30:27]: [0x7f2529cfc700 | 139797592000256 | ./main.c | process_post | 348] File [0x7f2529cfb820|139797591996448|"/home/anon/collectors/4chan_results/h/3807816/1412670867551.jpg"] does already exist and won't be downloaded anymore ...
[2014-10-07 19:30:27]: [0x7f2529cfc700 | 139797592000256 | ./main.c | process_post | 350] Returning: 1
[2014-10-07 19:30:27]: [0x7f2529cfc700 | 139797592000256 | ./main.c | process_post | 280] Entering with itf: [0x61d5a8|6411688]
[2014-10-07 19:30:27]: [0x7f2529cfc700 | 139797592000256 | ./main.c | process_post | 311] Response base: [0x7f2520101c60|139797428444256], current position: [0x7f252010b304|139797428482820], response length: [0xa85f|43103], bytes left: [0x11bb|4539]
[2014-10-07 19:30:27]: [0x7f2529cfc700 | 139797592000256 | ./main.c | process_post | 341] Took care that the file string for us (length: [0x3f|63]) is saved in the current stack: [0x7f2529cfb820|139797591996448|"/home/anon/collectors/4chan_results/h/3807816/1412686028437.jpg"]
[2014-10-07 19:30:27]: [0x7f2529cfc700 | 139797592000256 | ./main.c | process_post | 348] File [0x7f2529cfb820|139797591996448|"/home/anon/collectors/4chan_results/h/3807816/1412686028437.jpg"] does already exist and won't be downloaded anymore ...
I open a console and enter in
[code]
var link,imgs = document.querySelectorAll(".fileThumb");
document.body.innerHTML = "";
for (var i = 0; i < imgs.length; i++) {
link = imgs[i].getAttribute("href").split("//")[1];
document.body.innerHTML += "<a download href='" + link + "'>" + link + "</a><br />";
}
[/code]
then in a terminal
nano urls, copy over links
mkdir a && cd a && xargs -n 1 curl -O < ../urls
is there a way I can make the web browser via js download the files?
>>3814610
Can't find DownloadAll on Google chrome store
>>3817008
Because it doesn't need Javascript.
Because it auto-updates everything in an interval that you can set yourself.
Because it supports parallel downloads.
Because it uses the API of 4chan.
Because it uses If-Modified-Since to determine if there are new images.
Because it supports SOCKS4, SOCKS4a and SOCKS5 (withouth GSSAPI authentication).
Because you don't even have to be home and it will still download all images.
Because it is fast and efficient as fuck - monitoring 1000 threads? No problem, it's I/O and memory bound, not CPU-bound. The latest version is running 2 days now and it barely took 10 seconds of raw processing time of one core. And that is over SOCKS4a (same stuff Tor uses).
Because right now it uses 0.1% of my entire memory.
Because it's so much damned optimized that I do not deal with milliseconds or microseconds, but ticks.
Because you don't need a browser for it.
Because it switches user-agents per thread.
Because you can add new threads any time you want, the program will include them in the next iteration.
Because it supports load balancing and lets threads connect to different IPs of the name domain.
Because it guarantees that only images that were uploaded in a post get downloaded. Any error images/captchas/advertisement/tracking images are skipped.
Because it's just 39 KB in size, and I didn't even optimize it for size (most of it are strings for User-Agents and debugging prints).
Because it already downloaded 17 GBs for me, and I haven't even tried to be hardcore.
Because it's written in C.
Because you cannot program.
Because you are an idiot.
>>3817061
Addition:
Because it removes threads from its list that 404'ed automatically.
>>3817061
Another addtion:
Because I've already written an API for removing metadata from PNGs, JFIF and EXIF files, and I am planning to integrate it into the program.
>>3817061
And another one: Because it uses persistent memory for certain tasks and always tells the server it supports gzipped encoding.
Because it does not try to track you, but rather tries to avoid it.
>>3814700 posted shit, because the Perl libs have HTTP headers that makes it clear that it's a bot performing a task. My program fools the server that it is a human being through a browser.
Because I am maintaining it, and I am right now trying to optimize it (for speed) even more. My next goal is to get the SOCKS handshakes done faster.
Chrome Users: https://chrome.google.com/webstore/detail/download-master/mcceagdollnkjlogmdckgjakjapmkdjf
>>3817068
>Use API
>Pretends not to be a robot
>Not absolutely retarded
>>3817074
Since fetching the JSON file and fetching the images does not have a viewable linkage to the server (API and image server are different ones), the program does not need any cookies, and it also changes HTTP headers ... yeah, absolutely retarded, trying to fool the server it's not a bot when downloading the ima- oh wait.
I laughed hard.
>>3817077
>3817077
jesus christ you're dense.
If they have an API, and it's documentation has image file location, it means they don't give two fucks. whatever makes you feel like 1337 i guess
>>3817068
I think you might be the king autist on 4chan
>Inb4 another 10 page essay full of angry nerd ramblings
>>3817061
interesting, do you work as programmer or are you a wizard?
>>3817081
Heaven, are you dumb.
The file links are the very same normal people use to download them manually. The JSON file is downloaded only to build them up, because otherwise you cannot know them. And the JSON file is needed because these days 4chan blocks accesses to the board server from Tor.
I sat there two minutes just facepalming about your stupidity. You don't even know shit, but keep babbling about things your mind isn't even big enough to understand.
>>3817084
Thanks.
>>3817087
First one.
>>3817091
I'm guessing you are from the US? How old are you? Do you hang out on /g/ or do you think it's full of consumber retards?
We only had a little C at college, how do i git gud?
>>3817092
In fact I am from middle europe, in the beginning of my twenties, I don't hang out there because I'd just rage about them apple faggots, and if you wanna be good, you need to enter what I call "autist meditation". Put yourself into shit that others told you you don't have to do - like, programming a downloader for this board. Write an HTTP parser. Do stuff that has been done before, and try to even improve it.
Example: my self-written HTTP parser takes 2000 ticks to do one syntax checkout. There are HTTP parser, but there are no ones that fast.
Also it helps a lot to read other people's sources - so that you see NOT how to do it. Most programs out there are broken beyond repair in my opinion - for example reserving just 1024 bytes on the stack for a path, while there are file systems which support even bigger paths, and then using strcpy to copy them. Vulnerability detected.
You don't learn to program in college or university, but in real life.
>>3817094
You're not german are you?
>>3817201
Could be. Or Austrian. Or dutch. Or Belgium.
Middle Europe is a wide area.
>>3817236
One deserves to have an inflated ego, the other one does not. I deserve it, you not.
It's just that simple
>being this proud over a web downloader
>being this proud over the most awesome web downloader written from scratch
Something you will never achieve in life.
>not making your own 4chan thread downloader
>pic related
>>3817282
Holy shit, you are an asshole. Whatchya trying to compensate for?
>>3817061
0.1% of your memory is still quite a lot. If you're on an 8-gig machine, that's 8 still MB. I don't do much web development, but I do mobile development and can tell you I've written entire applications that are less than that.
Just don't be a dildo. Open your shit to the public.
Protip: I've made more money by making free and open-source applications with "donate" options than I have with paid applications.
jdownloader works great for me.
I'm working on one right now as we speak. I got it working, but its in Java and really shitty. You can have it if you want, I'll post it on GitHub.
>>3817061
What a faggot.
>>3817342
The reason for that is quite simple, if you think about it:
- each 4chan thread is processed in a POSIX thread. And each POSIX thread requires some memory for the thread's stack. Right now this size is 32 KB - do the math for a helluva them.
- also, each thread requires some memory on the heap, currently 4 MB, and with the option that it may be quadrupled on demand. I tried to use alloca instead of malloc and free whenever possible, but you cannot do that for huge amounts of persistent memory (because at some point you'd run on a stack overflow, and I didn't want to set a bigger stack frame in the makefile - it just don't feel right). Persistent memory means: the same memory that is used for saving the temporary result of a request is also used for decoding the result (because if you ever looked into zlib you'll see that it works best if you got enough memory so that it can perform its task in one processing call).
- also my program does not know partial downloads. Either it got downloaded, or the connection was reset and needs to be reestablished , or it fails. There is no 80% of an image - and the 100% of an image on success is accumulated in the memory, not on the drive.
I once even had a perl script that would do the job for me, but they never got the thread implementation right. 30 threads would require 8 GB and 20 percent of one CPU core. So if you compare the current stats, it's quite impressive.
>Just don't be a dildo. Open your shit to the public.
Dream on. Such programs are worth tenthousands of dollars, and I am supposed to give it away?
>>3817359
Uh, java is indead crappy. I want to learn it right now, but it just stinks against C. It's 50 times slower than the C equivalents and hogs up so much memory I thought I am back in my Perl times.
>>3817320
>>3817363
U mad? Prove you are worthy, then we may talk about it. Or continue to be mad and serve my amusement. European masterrace is laughing madly right now.
>>3814561
http://pastebin.com/VsN19KmZ
name it Grabber.java
save the thread as chan.html
output will be gifs.html
shitty but it works
if you want to change which board it works for, just change the two "http://i.4cdn.org/h" es to whatever board you want.
Also it doesnt download them but with some tweaks to the program it can easily be changed to do so, I made it so they all show up on one webpage.
>>3817427
You are one of those people I am not sure of but certainly hope you are trolling. If you really are this proud of a program to download porn with and expect tenthousands of dollars for something other people have put online for free then I just can't help but pity you.
>>3817428
While this is certainly impressive I am not gonna pay money for it. Imagine I just want to download some nice manga dump on /h/ and nothing else. In that case it is far easier (and cheaper lol) to just run one of the alternatives posted in this thread. The Perl one might not be super-1337 like yours but it actually works well enough.
>>3817454
And again I just sat there, unable to think for a minute just because I couldn't bear the pure stupidity of your post.
I already pointed out all the programs advantages of common downloaders. Especially the reliability that images either get thrown away or are 100%-ly saved on the hard disk is something that even modern browsers cannot come up with (and believe me, I know what I am talking about - I have read the source code of firefox, and it's HTTP stack just sucks).
But hey, I don't expect anyone to pay me. If you want:
- slow
- crippled
- java-based (or generally non-native)
- manual
- non-updating
- singlethreaded
- hack-together
- memory and CPU consuming
solutions, I am fine. Besides the core elements of the program are all put into a separate library which is used by other crawlers made by me - so it's just not this program, but a hole bunch of them.
Your choice, mate. Enjoy the pain updating all threads you have an interest in.
>>3817457
That is of course true, and I cannot really argue against your reasoning.
But maybe someone has more requirements. I can just offer and describe my program. If you are not interested, that's OK.
(But if you are interested, but expect me to waste all the time I put into it by offering it for free - fat chance).
>>3817443
>importimportimportimportimport
>trycatchtrycatchtrycatchtrycatchtrycatchtrycatchtrycatch
Java truly is the worst language ever
>>3817460
I get that but I think you need the right tool for the job and if you just want images from one thread once every blue moon the script suffices. I admit though that if you wanna monitor a gazillion threads your program does take the cake.
>>3817461
>I had a problem so I used Java, now i have a problem factory.
>I Java truly was garbage collected, it would collect itself
>>3817459
Is this the first time you post about your program on 4chan?
>>3817470
Indeed it is.
Why do you ask?
i created this http://sourceforge.net/projects/xchanthreadimag/ a while back but it broke with the update
i fixed it and still use it but was too lazy to upload an updated version in case of hidden bugs
i'll upload the new version later today if people still need a program
>>3817472
>Due to .NET framework being shitty and downloading empty images I added a checking function that auto deletes the file and re-downloads them
What did I say about broken HTTP stacks?
>>3817473
a half decent programmer can take something broken and implement a workaround or fix
and i'm not selling this. it was something i made for myself that i'm sharing with others
>>3817474
No, a half decent programmer thinks "into the trash it goes" and implements it correctly. That this isn't done often should actually worry you about the quality of the software surrounding us.
Working around things is what idiots like Poettering or Drepper would do.
I guess they pay you fine for sharing it with others?
>>3817476
Hilariously enough 99% of "programmers" are absolute shit. At my college they are actually removing a lot of coding related classes because people complained they are too "hard".
Go figure what that means for the future (did someone say influx of security holes?)
>>3817480
No surprise for me, really. Because I already wrote:
>You don't learn to program in college or university, but in real life.
But what would I know. I am just a lowly C programmer that ditched university.
>>3817480
That's what happened in my final honors year; all the management-to-be cried so much the tutor handed out source code worthy of a pass.
Easiest A of my educational life, but quite distressing.
>>3814578
Hate to sound like an idiot anon, but how come all the pictures I save with this come up in tiny resolutions and are impossible to see? Is there some setting or filter I have to change so they all save at their posted size?
>>3817494
Because I think it just searches for the HTML img tag. Not for a hef.
Classic.
>>3817482
Do you have any public code one can study?
If not, how about some ressources to learn network coding in c?
also do you prefer
if (foo) {
or
if (foo)
{
>>3817502
There are lots of open-source programs, mate. The X-Server and mednafen need some cleanup (although the second one is coded in C++). But the Code for the Server is beyond all hope so that I decided I will only audit it for hard cash, and the faggots for mednafen didn't want my fixes (no answer at all, not even denial) - so I guess fast-forwarding will continue to cripple users sound output.
Otherwise there was once a tor library available in the Onion network which I downloaded back then. The link is:
am4-wuhz-3zif-exz5u
Remove the '-', add a '.onion' to the end, and put the link to a browser which is configured to use a SOCK4a Tor-Server for DNS resolving (a.k.a the Tor Bundle - google it).
And I prefer:
#include <errno.h>
#include <stdlib.h>
int main(void)
{
int merrno;
if((merrno=load_file("4chan.conf")))
{
exit(EXIT_FAILURE);
}
exit(EXIT_SUCCESS);
}
>>3817504
>The X-Server and mednafen need some cleanup
>X needs cleanup
X needs to be abolished
Wayland is the future, anon
>>3817505
Normally I would agree. But there are systems that are running with X for some years now and will continue to run it, and I cannot imagine (not with the average skill of our future administrators who are overchallenged with simple Windows server installations) that this is going to change in near - or far-away - future. In fact we are lucky if those admins even apply the newest versions of certain packages (I won't give names, but I know a company that deals with huge amounts of personal data in my country, and they applied the OpenSSL patches 4 weeks after Heartbleed came into public focus. And they were actually rather fast doing so). Wayland supports a X Compositor which - if the rumours are true - runs even better than native X, but ... well, never change a running system, right?
>>3817509
but anon if the system runs fine why change it?
captcha is hebrew are you kidding me?
>>3817511
In fact it does not run fine. We are just told so. And it does not run fine for three reasons:
1. Incompetent programmers.
2. Insufficient documentation (see drivers on Linux).
3. Lots of workarounds for problems that should have been abolished years ago.
>captcha is hebrew are you kidding me?
4chan is the jew.
>>3817513
Is your program cross-platform?
>>3817515
Cross-plattform for POSIX-compatible plattforms right now. Not yet for Windows, but am working on it right now - of course I could add a wrapper to it (Cygwin), but that's bullshit if all I need is are some API calls wrapped around a preprocessor layer.
>>3817517
At work we had a posix dependant program which ran fine on windows with just one simple dll
>>3817520
That is Cygwin then: www.cygwin.com.
But I do not want that. I want my programs to run natively on a plattform. Otherwise I could just switch to Java.
>>3817523
Do you have a thread per 4chan-thread and the downloading is sequential? Or do you also multithread the downloading of images from a given 4chan-thread?
I am asking because I am not sure if there is a significant increase in performance (read: speed) when using multi-threading on a normal drive.
>>3817513
In your opinion who are some good programmers?
For example do you think any of these are good at what they do/did?
Bill Gates
Gabe Newell
Linux Torvalds
John Carmack
>>3817525
As I said: 1 POSIX thread per 4chan thread. The download of all images in a thread is sequential within this thread.
>>3817528
Bill Gates is a good salesman, but for his personal skill I just know the story how he managed to get the BASIC interpreter below 4 KB in size. That's requires some skill that I am willing to admit.
Linus Torvalds is a true C hacker. He puts certain requirements on code that's supposed to be checked in (the reason why Poettering isn't allowed to work for kernel code right now). But in my opinion he's not rigid enough - I had my time of kernel debugging as well, and many modules are just coded as if done by a student who does not understand what he is doing. There are no constants, no comments, nothing. Of course he's aware of that (he once stated that he does not sleep well anymore knowing there's no person who is able to overlook the whole kernel, and that the complexity is just going to increase, not decrease), but ... well, good programmers are rare.
About Gabe Newell ... I dunno much about him except that he's working for Valve. And although I applicate their effort to make Linux more compatible for games, in my opinion most problems would be solved with a better graphics stack in the kernel and with a patched-up X-Server. And I do not see how yet another linux distro would help to fix these problems.
And John Carmack - well, I didn't even know five minutes ago, but he programmed some games that I just loved as a kid (Commander Keen and Doom).
But I am just not sure about this oculus rift project. I mean, better graphics do not indicate better gameplay - in my opinion the best games were made in the 90ies. And thats no nostalgia, because lately I began playing games from that game on emulators that I haven't played in my childhood, and each and every game I played was more entertaining then playing TES Skyrim. Maybe I am already too old for such graphics, I honestly don't know.
>>3817562
Dein Englisch ist nicht schlecht für einen Deutschen. Machst aber typisch deutsche Fehler
>>3814561
bro just open all the pictures in the thread and right click save as
boom done all in a folder
ofcourse it saves extras like the banner and the thumbnails but its easy as fuck
>>3817569
Ich sprechen ein wenig deutsch. Aber das ist nicht gut genug.
>>3817497
So is there any fix then? If not, whats the point of this add on?
>>3817657
These settings make it work for me.
Give it a shot.
Is this thread a joke?
C is taught everywhere, it's basically a fundamental introduction to manual memory management and concepts like threading and sockets.
You've pretty much taken those fundamentals and you're presenting it like it's some arcane art.
You're self taught, be a little more humble.
>>3817517
>API calls wrapped around a preprocessor layer
Whaaat the fuck does that even mean.
And it works in Cygwin and NOT windows because you're using basic taught material ie: intro to linux.
>>3817779
Linux has socket.h etc that Windows does not. Go ahead and use Winsock for that with some snazzy platform header guards.
>>3815254
>ychan
THIS. works fine for me
>>3817657
What would I know? I did not promote it.
>>3817774
[] You have understood what this thread is about.
I did not promote the program for being written in C, but its features. Read them up; I have written enough about them.
>>3817779
That means that during compilation the compiler decides which headers to include, which libraries are linked in, which call is translated etc.
And I don't need Cygwin. Because this night I managed to compile my lib for windows. Natively. Without any additional stuff.
Thought I'd throw a contender in because fuck it.
There are millions of potential optimisations but this was written in no time so don't judge it too harshly.
Enjoy:
http://hastebin.com/abotopijuy.avrasm
>>3817822
Right after I finish it I noticed a mistake. Modify the regex to only use file links instead of file + image links.
Results in half the execution time.
Altered version:
http://hastebin.com/jebaxetiye.avrasm
>>3817824
Oh I should mention features right?
Java, so multiplatform as standard.
Source provided, so rest assured not a virus.
Multithreaded on a single URL, so images downloaded in parallel in a single thread.
No error mitigation or recovery outside of multiple unhandled try/catch, so expect debug to be tedious.
Possible improvements might included adding a simple Swing dialogue for grabbing the URL or clipboard copy then wrap the whole thing in a .JAR
>>3817289
Interesting. Post it on github? Make a thread on /g/?
>>3817091
>>3817061
>using C instead of C++11
>wrote a little tool that parses a JSON file and downloads porn
>thinks it's worth thousands of dollars
>tries to lord it over non-programmers on /h/
Wow. Just wow. I don't believe that you're a professional. Certainly not a highly paid one. And ALL of your coworkers, if you have them, despise you for the little shit that you are.
>>3817092
You can "git gud" at programming the same way you get good at everything: practice by writing trivial programs like he did. And, yes, it's a trivial program to write.
>>3817459
>Firefox's HTTP stack just sucks
Sure. Sure. The second most used browser in the world is a complete fucking turd written by amateurs.
>I know what I'm talking about
Sure you do. You fucking said so, didn't you? Why wouldn't we believe you?
>compares file downloading and io with the HTTP stack
So... you failed your first assignment of programming 1301?
>derides everything else as hack-together
>his downloader is CLI only
I don't pity you. I've nothing but contempt. Your autistic, vain, neckbeardy, incredible arrogance is just reinforcing the neckbeard stereotype.
I know for a fact that you're overweight, you've never had a girlfriend, your parents make fun of you or ignore you, and so do all your peers.
And, given the way you behave in this thread, I have no doubts about why.
>>3817912
>Sure. Sure. The second most used browser in the world is a complete fucking turd written by amateurs.
you mean third
This is at the same time the saddest but most interesting thread on /h/ I've ever read.
Also a friendly reminder that advertizing is prohibited in the rules. Humbly presenting your product while offering free preview / demo is little bit different than this.
>>3817907
God, are you mad.
I can feel your anger right here. Must suck being so stupid that you cannot come up with something in your life.
(side note: the co-workers are heavily impressed by the performance gains I got).
>>3817912
>Sure. Sure. The second most used browser in the world is a complete fucking turd written by amateurs.
Have I read the sources? Yes.
Have I understood them? Yes.
Have I tested them? Yes.
Do I know the weaknesses of the stack? Yes
Have you done all that?
Also, do you know how most projects are made? I will because I like to destroy little dream bubbles:
They just hack it together, and then the other programmers are too lazy and begin to workaround problems that were made in the initial implementation. Go on, read the sources, These comments surely are hilarious.
That goes for closed-source as well as for open source. OK, not entirely. In closed-source they do decide to re-write some certain stuff, but for backwards compatibility the old stuff is kept in additionally.
>compares file downloading and io with the HTTP stack
Oh wow. You are an idiot. Sorry, I mistook you for someone I could argue with.
>ad-hominem, mindless babbling
Yeah, whatever. While you were writing this post without any solid content I just finished the wrapper for Windows.
Your "pity" is just overdramatized, while in reality you are just angry that I finished something of value, but wouldn't share it with the "common people" that usually get everything shoved up your ass.
But go on. I am taking a break right now and could do with your clown performance.
>>3817766
holy fuck thanks you
>>3817805
You know your stuff to some extent but, and perhaps I followed the thread incorrectly, but why are you making such a big deal about getting c to compile in a Windows environment? Issue has always been the other way around for me.
>>3817954
User targeting. The library is already usable for POSIX systems, but it also needs to run on Windows.
And funnily I got more problems with porting it to Windows than writing it to Linux - Windows compiler sometimes behave strange.
>>3817934
Do you watch animu?
>>3817978
Why should I?
>>3817992
Then why do you hang out on /h/?
>>3818023
First time on 4chan for ages, and the thread just passed by.
To the C guy. Why are you reinventing what existing libraries can already provide?
Pair that with the fact that other libraries will also be strongly rooted in the standards they handle due to years of criticism and improvement.
Was it a learning exercise or are you just trying to drive down the binary size?
Seems like wasted time to me, looking for your reasoning.
>>3818038
I'm more interested in why he has to be such a dick about it
>>3818038
You are an idiot.
My program works better than established libraries because i am a genius and you are not
>>3818038
1. These existing libraries are either overloaded with stuff you do not really need or are slow as fuck, or are erroneous or all of the three.
2. Of course I also did it to learn about writing APIs, writing reliable network socket algorithms, and trying to explore the boundaries of C. And there are few of them.
3. I also wanted a reliable library for all purposes that a normal programmer could encounter. For example, there is no native way to get the difference between local time and GMT, or at least Stack Overflow did not find one. So I implemented a call for it.
Another example: the calls for mapping a file into memory on Linux and Windows are pretty much the same. Wouldn't it be nice to call just function and have, let's say, an PS1 image right now in your memory?
Normally to do that, you'd have to call open/CreateFile and then mmap/CreateFileMapping. You'll have to remember that you cannot just close the file and free the file's content, you have to unmap the file, again with the corresponding API call. Oh, and I forgot to mention proper error code checking.
What if you have a large text and want to search it for a substring? Or if you want to save the addresses of a beginning and an ending tag in that text? What if you want to do tick-based time measurements? What if you want to connect to a SOCKS service? What if you want to print out the size of a file without loading it in or using FILE structures/ifsteams? With my library it's two functions, they work on Windows and Linux, and I don't have to care for anything else.
What if you want to write/read a digit or hexdigit (important when programming an HTTP stack), but you do not want to use sscanf/sprintf because they take up to 50.000 ticks per call? My functions take up barely 700.
What if you want all of that at once?
BTW: my HTTP stack is RFC7230 compatible - the newer RFC for HTTP. Most old stacks will still rely on 2616 and 2145.
>>3818042
Stop to act as you were me. I am rather civil as long as you are civil to me and don't post bullshit you do not know about.
What if you rely on functions that are available only if you put preprocessor definings into your code? Fun fact: the original memmem implementation does not even care to implement a boyer-moore search algorithm (that's about existing libraries tend to do better jobs), and you have to put
#define _GNU_SOURCE
before you include any header files. Thus I implemented them all in one batch. And depending on the parameters they are up to 20 times faster than the original functions.
I have lots of programs that require these functions, but I want to have a unified interface for them. You see why I put so much effort into it?
>>3818040
I do not try to be a dick about it. You ask me a question, I answer, rather nice as I'd guess. You doubt my competence or insult me, I hit back.
And wanting to get paid for good work does not indicate that I am a dick. You are not interested, fine - I already wrote like 20 posts ago I'll have to live with it, and I stopped advertising my work. But if people ask me questions, I will answer them. Why shouldn't I?
>>3818054
Thanks for your insight.
You should however be aware that /h/ is essentially a gathering of people who aren't too keen on being separated with their cash so don't expect to make a reasonable sale.
If you've put such a large amount of effort in to software then consider shipping it as part of a utility instead of keeping it specialised.
>>3818054
Why C instead of C++?
i have used this a lot.
sourceforge.net/projects/fourchan-dl/
>>3818942
I tried not to be a dick here, but the quality of the code clearly shows that this program indeed was put together.
>QTextStream in(stdin);
>QTextStream out(stdout);
Then just use cout and cin, that what they are for, dammit.
>// Just checking the version -> close immediately
> return 0;
> }
> else {
Condition that returns the function, but then "else"? ARGH.
>ImageThread::getNextImage
Just search for the function, there's enough bullshit in it to make you fringe.
Or this: >ApplicationInterface::ApplicationInterface
And the thread handling code just calls for race conditions. Also there is nearly no debugging code.
You were using this? Impressive.
>>3818940
I did my share of C++, but gave up after I found out just how messed up object-oriented this code is. I don't mean object-orientation in general, because C also has objects. The only difference here is that you cannot have your classes inherit code from other classes/structures, but it is possible.
C++ might be slightly easier, but you pay for it. I didn't want that, so I wanted to see what I could do with plain C. And there is not really anything that I miss about C++.
>>3817494
There are two alike images; the thumbnail and the images they refer to. Make sure to have DTA set on links, sort by downloads, and highlight and exclude all images that end with s (ie. 9807089890s.jpg). This will make it so that you only download the referred image.
I use this free python package
https://pypi.python.org/pypi/BA-4chan-thread-archiver/0.7.0
>>3818987
You should post some of your code. I'd really like to see it after all of this
https://github.com/SuperGouge/ChanThreadWatch/releases
>>3819484
Now this is some well maintained and well written code. Even my C App can't compete with that.
>>3819484
I like how the parser doesn't use Regexes, would save a lot of time but annihilates flexibility unless you match broadly, the problem being broad matches result in false positives. Good job on that manual parse, I'd never have the effort for that shit.
>>3819487
Indeed. The best language is not the fastest or easiest but the one that gets the job done. If you guys don't already know there's a bunch of existing tools to clean and refactor code such as CodeMaid and Resharper. Using these you can consistently adhere to rules and produce concise code.
>>3817061
pls somebody buy it and put it on pirate bay, this faggot doesn't deserve money.
>>3819512
Not even worth the effort.
>>3819487
Nice try, dude.
First you are not me, second this code is a fucking mess. MUTEXES? Are you serious?
I haven't used ANY mutexes in all of my code, and it runs fucking fine. There are applications in which mutexes/futexes make frigging sense - this is NONE of it.
Also:
>[STAThread]
Yeah, right.
>>3819402
What should I post there? Too short, and it would give anyone a clue WTF I did there - I might even have used it for another purpose. Too long, and I'd reveal crucial informations.
Sorry, can't do. Honestly wish I could do, just to show how this stuff is done.
>>3819492
You can do that even without Regexes. It's called sscanf.
Also, that last statement is not true. Code beautifier can just hide the ugly mess, but it can not optimize it. It cannot get rid of these mutexes.
>>3819512
Yeah, very intelligent, considering that all my customers would be listed in my database and would be held responsible for breaking my copyright.
Also, who'd be stupid enough to pay for it, but then give access to everyone for free? And last, but not least you would not get any updates, because you guys just would have the binaries and cannot change the sources.
>this guy arguing on the internet
>on an anonymous image board on the internet
>in a hentai thread on an anonymous image board on the internet
>with other anonymous people who don't give a shit
>for almost a week
Do you not have something better to do, o almighty codefag? Unworthy are we commoners to be in your presence.
Post that code, if you're so great.
>>3819657
Ever heard about multitasking? I am testing my software - especially the HTTP stack - right now. Also, I am currently waiting for an answer from some other C programmers how to optimize certain stuff.
>Post that code, if you're so great.
Hahaha, o wow.
YChan Downloader still works.
>>3819648
Mutexes are pretty much essential when dealing with a limited thread pool.
If we push our list of images on to some queue then we can dequeue them in a threadsafe manner without dupes/skips/corruption. Assuming you're not using mutexes then its wholly possible that your data structure is, you need some kind of blocking for true multi threading, otherwise you're going to have to prepare the threads with the data they need beforehand.
Didn't look over the C# but considering how it does the whole winforms thing I can see why it would be appealing.
Doing things without regexes is indeed possible but I'd have probably taken that route too. It's just so easy to pattern match.
I don't know why you're still considering selling C code on /h/ but good luck etc.
Work is work, but I wonder how much value others place on it when so many alternatives (open source too) are available. Aim higher than /h/.
I might rework that java code to use an arbitrary pool, creating and destroying hundreds of threads can't be good.
>>3818987
I think if you are careful C++ doesn't have any inherent price you have to pay. Of course if you make rampant use of some of its "features" you may end up with shit performance compared to C
>>3819741
Not the C guy but in general you want to reuse your threads as much as possible. You pay a considerable overhead every time you create/destroy threads.
Thread startup in specific can take some time. So just create a pool and assign work to threads as they finish
>>3819753
I am not going to post code, but I will explain how I solved the problem.
I said at the very beginning that I take a file in which links to all thready that you are interested in are taken, and that I start a thread for each link that I find.
Well, I am also dealing with limited thread pools here. Three storages in the memory are important here:
1. the actual thread pool, which holds the association between custom thread ID and POSIX thread ID. The first is used so that each thread knows which resources were assigned to it, the second is later used to clean up all kernel-related data (POSIX threads are hybrid threads, scheduled by the kernel, and therefore they can be run on different processors parallelly).
2. The threads stack, which is actually allocated from the heap and put in one solid piece (at least from user-space perspective).
3. The parameter pool, which holds all informations what the thread is supposed to do.
These memory blocks are reserved and initialized before any thread is started, and they are not freed, but used until the number of threads to start change. In that case, the old memory is reallocated (either more or less memory for the thread pool). This is done at the same place at which the initial pools were allocated - BEFORE any thread is started for the current iteration. So, there is actually no NEED for mutexes.
>I don't know why you're still considering selling C code on /h/ but good luck etc.
Thanks. But actually I am just looking for a worthy competitor to beat.
Don't worry, so far there has not been one.
>>3819752
I have yet to make some tests here. It might be true that, if you are careful, you can have the same performance as with C. But I doubt that, considering that even the linking process is a different one (to supposed function overloading).
>>3819765
How many units of your program did you sell so far?
>>3817071
Does this thing have a folder directory option? I don't want to downloadd 500 images and have my desktop suddenly swamped
>>3819812
you should really use pastebin
>>3819813
Sure thing.
v1.03
Cleans up temp folder and shuts down threadpool.
http://pastebin.com/embed_iframe.php?i=ngC2xjGw
>>3819819
v1.04
Runnable JAR that requests thread URL in a popup window.
Removed print statements.
JAR:
http://a.pomf.se/cmddaa.jar
Source:
http://pastebin.com/embed_iframe.php?i=Ergvpq29
>>3819825
v1.05
Download class is more flexible now, uses provided paths.
Some cleanup of unused stuff and minor refactoring.
JAR:
http://a.pomf.se/ksmsqn.jar
Source:
http://pastebin.com/embed_iframe.php?i=Hw08VqxT
>>3819765
Can you explain more about how to write your "own network stack"?
googling that gives me stuff like "writing your own internet protocol" etc
>>3819846
Why, yes, I feel like it right now.
Most important thing to know is that if you want to read from a socket, you are doing this on TCP/UDP level. That is, the kernel has already stripped away the header fields and passes the payload (that is, what you sent/what you got sent) to user-space (that is, your program).
So basically, you just "send" your data:
#define MY_PAYLOAD "GET / HTTP/1.1\r\n\r\n"
int my_send(int sock)
{
if(!sock)
return EFAULT;
send(sock,MY_PAYLOAD,strlen(MY_PAYLOAD),0);
return 0;
}
You asked the server (in a very simple way) for the resource "/". If you are lucky, you are going to get it now. But how are you going to receive it? For that, you "recv"eive everything that is sent to the socket:
#define RESPONSE_MAX_LENGTH
char*content=alloca(RESPONSE_MAX_LENGTH);
if((cur_pack_length=recv(sock,content,RESPONSE_MAX_LENGTH,0))<1)
{
return ECONNRESET;
}
But recv has some small problems: it just gives you what came it, if it's too big, you won't get everything, and it blocks if there is no data waiting for you. And how would you know if the server sent you everything that you wanted to have - like an hentai image? That means that each and every time recv returns you have to perform a check if the data that you wanted to have was already sent to you, in terms of the rules that the protocol (mostly HTTP) define. And when everything was loaded in, you want to have that content in an easy-to-use structure.
And another problem with recv: What if the connection suddenly dies? I just had this for testing purposes, and right now the downloader got stuck:
>>3819858
why not just:
char content[RESPONSE_MAX_LENGTH];
instead of
char*content=alloca(RESPONSE_MAX_LENGTH);
[2014-10-14 21:58:18]: Trying to download file: "/h/1407954177629.jpg" for file: "/home/anon/collectors/4chan_results/h/3783128/1407954177629.jpg" (Size: "29045", Resolution X: "149", Resolution Y: "194") ...
[2014-10-14 21:58:18]: Downloaded file: "/d/1412889496799.jpg"
[2014-10-14 21:58:18]: Trying to download file: "/d/1412889534247.jpg" for file: "/home/anon/collectors/4chan_results/d/5758222/1412889534247.jpg" (Size: "2355788", Resolution X: "1400", Resolution Y: "1980") ...
And it just stays like this forever. Until the kernel is nice enough to inform user-space that the TCP timeout finally came - which can be quite some time.
I won't get into too much detail here, but before you go pass the thread control to recv, you want to call select with an appropriate timeout value (for SOCKS based connections I gave 90 seconds before it hits the kill switch). If select hasn't got any packages within that time, it just kills the connection and returns:
FD_ZERO(&readfds);
FD_SET(socket,&readfds);
savefds=readfds;
mtv=(*tv);
readfds=savefds;
select(max_socket,&readfds,0,0,&mtv);
if(!FD_ISSET(socket,&readfds))
{
return ETIMEDOUT;
}
if((cur_pack_length=recv(socket,memory,memory_length,0))<1)
{/*...*/}
overall_length+=cur_pack_length;
/*And after that you have to to a synatx checkup for everything that is stored in memory*/
if(http_check(memory,overall_length)!=N_OK=
{
return EPROTO;
}
You want to have http_check to be as fast as possible, because in the mean time you cannot recv any more packages.
And you want to have a function that parses the values in memory and put them into a nice structure, so that you can access them like:
if((merrno=http_stack_request(itf->itf_socket,*(itf->itf_stack_thread_request),itf->itf_stack_thread_response,&defaults.threads_socks_timeout,itf->itf_ih)))
{
DEBUG_PRINT("Could not send HTTP request for JSON object,error code: %i\n",merrno);
return (void*)T2I_JSON_FILE_NOT_FETCHABLE;
}
You get the concept?
>>3819862
In that particular example, there's no real difference, and even harmful, because alloca does not exist in Windows environments (they got _alloca instread).
But what if RESPONSE_MAX_LENGTH wasn't a static constant, but a value you just read in?
#ifdef WIN32
#define MY_ALLOCA(x) _alloca(x)
#else
#define MY_ALLOCA(x) alloca(x)
#endif
size_t a;
printf("Please gimme the number of elements you wanna store: ");
scanf("%d",&a);
/*This does not work then*/.
char memory[a];
/*That would work:*/
char*memory=MY_ALLOCA(a);
>>3819863
>mtv=(*tv);
>>3819865
True, if you store the max length in some config file and read it then it obviously makes sense
>>3819858
> send(sock,MY_PAYLOAD,strlen(MYPAYLOAD),0);
> return 0;
Some friggin' programmer you are, mate. You're not even checking the return from send(2) to make sure the amount of data you've sent is the same as the contents of the buffer. Also, yer not checkin' errno. Failprogrammer is fail.
>>3819873
Oh, that was just hacked in. The real function looks like this:
MERRNO_TYPE my_send
(
const int socket,
const char*request,
const size_t request_length
)
{
MERRNO_TYPE merrno=0;
ssize_t cur_package_length;
size_t sent_bytes=0;
while((cur_package_length=send(socket,request+sent_bytes,request_length-sent_bytes,0)<1
&& (sent_bytes+=cur_package_length)<request_length));
cur_package_length<1 && (merrno=errno);
return merrno;
}
>>3819887
Argh, little error while erasing the symbols: It would be >0, not <1 in the while header.
>>3819835
I'm just trying to help OP and share some effort, my goal is not to show off, sorry it seems that way.
Anyway, isn't the point in this thread to find thread downloading tools? I think I'm relevant.
>>3814561
img2tab if you are using firefox.
>>3819890
Does your app download webms and gifs?
>>3819996
It does GIFs, if you add the extension in for webm it will do those too. It's a one line addition.
