I've been prototyping with an international news aggregating service. I'm not a web developer nor a usability expert so any comments and improvement suggestions are much welcomed! Currently supported countries are US, UK, DE, RU and FI. Update cycle is 15 minutes. The topic aggregation is not perfect but works fairly well.
http://spidr.duckdns.org/
Perfect, I heard someone talk about how they wanted this a week or two back, seemed like a good idea. need to share this with /Pol/
>>20054
Looks like a pretty good idea, if you keep the URL the same i've bookmarked it and will be using it as one of my websites i go to for news.
>>20054
This is really great! Thanks for sharing, OP! Cheers. I'd post congratulatory tits&ass pics if I could.
inb4 /pol/ ruins this somehow, even if just by making it crash from too much traffic
>>20054
>Last update: just now ago.
will this be a new meme now?
>>20054
why do you only have left-wing news sites there?
>>20086
It officially is as of just now ago.
>>20087
I really don't think he was trying to do that on purpose, I think in the interest of being a true aggregate he included everything available and most of those just happened to be left or center left sources. After all, sites like Forbes and the WSJ are mostly paywalled.
>>20094
or it could be that he simply didn't want clickbait like breitbart and daily mail in there
>>20086
>/news/ first meme
Took ya'll long enough.
This is pretty great OP.
If it isn't asking too much could you make this integratable into an rss feed? Though I don't know how these things work so if it's too hard forget about it.
>>20054
You are the new king of /news/, OP, today you were not a faggot.
More people should make things like this with other newssources. This is wonderful
>>20054
Bretty gud, is one america news a source in that? They're pretty reliable.
>>20054
First time I see something like this.
Could anyone explain what a news aggregating service is?
>>20125
A news aggregate is a site or service that takes all possible news stories on a subject or in general. It's what drudgereport pretends to be, googlenews tries to be, alex jones's infowars wishes it was, and what alt.news on usenet used to be a long time ago.
Nowadays they tend too look like giant RSS feeds on paper or fancy ones like OPs or popurls.com break them down by general subject or other catagories.
OP here. Thanks for all supportive comments, they made me realise this is a project worth of further development.
>>20087
The intention is to include as many sources as possible so that people could see the same news from all political viewpoints and make their own judgements. I'm not very familiar with american news scene so I listed only the ones I knew. Ideally also marginal news sources would be listed.
>>20113
I will try to think how this could be done.
>>20120
One America sources "All", "US" and "World" now added.
>>20126
I see. Thanks for explaining.
I'll put OP's one on my homepages and see how I like it. Seems good so far.
Mods should sticky this thread. It could only help the board. I don't know how many other people are using this right now but it's neat to have our own little aggregator right now thanks to OP. I only hope OP isn't worried about bandwidth if some big board or reddit gets a hold of it via some crossposter or something.
>>20130
>I only hope OP isn't worried about bandwidth...
This is definitely a concern to OP. I've set an alarm that shuts down the service if traffic exceeds a specific limit.
>>20095
>huffpost
>not clickbait
ok.
Do you mind sharing the code OP? I ask so I can self host it if yours ever goes down. I won't start my own to compete with yours.
>>20146
Huffpost at least labels their opinion pieces as opinion. Can't say the same about some others.
You're right though, any site that Andrew Breitbart had a hand in the invention of is shit, Huffpost included.
>>20417
Thanks for caring, let's see how the traffic grows in the following weeks. The web front-end is running on Amazon while the aggregator itself is crunching in my living room. So far the traffic has been moderate.
I listed the current news sources here: http://spidr.duckdns.org/sources.html
Feel free to suggest new sources.
is the site down?
ist doesnt work
>>20191
Apparently my hashing function made too much assumptions and in some cases the aggregator crashed. Now it's fixed, thanks.
Looking good! Any way you could provide all the same in rss-format as well?
>>20203
Fool me, didn't read first. Anyway, +1 interested about the rss
I like it
It would be nice if the reader could expand or collapse the subheading under the main headline with a +/-; some headlines have 10-15 subheadings, many of them repetitive.
Other than that, great job!
>>20174
Option to select news sources?
>>20219
It wouldn't be an aggregator if you picked your news sources. That's the point, it's all of them.
>>20054
Looking good.
>>20174
add aljazeera
>>20054
nice. I just made this my homepage
>>20321
Make them optional. Or don't add at all. In case the reader interested and want to read/see more, there's multiple links for that there.
The pics were a nice touch, but it cluttered everything too much behind the text like that. And aesthetically, the transparency mixed them with the grey color pallette and just made them dull. I don't think you can win with text over pics.
It might be too much, but I think you best shot at it is showing only a stripe of the image at the top of the news-box.
Auto-magically fitting the image so it looks good (specially if you only show a stripe of it) is not a trivial problem. If you like the idea, consider bookmarking this script I found around. Maybe you can tweek it to look for a stripe:
https://29a.ch/2014/04/03/smartcrop-content-aware-image-cropping/
I hate to be the "and mobile?" guy, but this page is so lightweight, clean and to the point that it's perfect to read while travelling, so please consider making a one column version for mobile devices. Mobile is another good reason to make pictures optional, most of the world still has shit coverage.
The "+XX" buttons are great, and thanks for AJ, you are pal
>>20054
I love the layout, please for the love of god, don't fill it with shit tons of stupid images or overly large graphics.
Do you have a contact email? Create an account for the site only, and use it for contact, feedback, and eventually Paypal donations.
I'm not a web dev, but I'm sure me and others will help you if you threw us a mail adress.
Anyways, in the css block of ".cl span" (the mouseover preview), you use "visibility: hidden;"
This hides the element from rendering, but still affects layouting. The lingering ghost results in the page being wider than it should, and screws the positioning of the header and also makes the horizontal scroll bar to appear below.
Try with "display:none" instead. Comparison:
http://www.w3schools.com/css/tryit.asp?filename=trycss_display
>>20335
>>20341
>>20351
Okay, so pictures are probably not worth the effort. Although pic stripes sound quite sleeky. I know mobile is important, we'll see.
>>20375
Thanks, this was easy. I believe some articles still contain javascript that I fail to parse out. It's not executable code but it results in long continuous text that doesn't fit in the default box width. Also, I added a contact e-mail in the page source code.
OP you gonna get a better name than SPIDR right?
>>20351
100% this! Pictures are simply a waste of screen real estate; please don't add.
>>20491
I don't know, I'm starting to get used to SPIDR. It's better than SPAIDER.
Today I added a new theme concerning the U.S. Presidential election. The idea of themes is to provide fast access to current events. The theme view shows news from all countries, just in case some details or viewpoints are 'forbidden' within one country's media but open in another country. Too bad Google doesn't provide a free translate API anymore, it would've been perfect for this purpose.
>>20555
Not the other guy but I think you could do worse than SPIDR. Sounds like a space probe or something to me.
The themes are very useful. Don't worry about the translations, we can always do that ourselves if need be. Would it be a major revision to let users choose their own custom themes? (as long as they aren't saved anywhere?) Would this be bad for CPU load or bandwidth?
>>20574
>we can always do that ourselves if need be
OPs right though since it'll be much nicer for usability to translate headlines.
OP could use Microsofts or something though as a last resort. Twitter gets away with it.
>>20398
also, in ".col" replace
>width: 33%;
with
>width: 33.3333333%;
that 1% you were loosing shows up as 19 whole pixels of asymmetry in a 1920 monitor.
It makes sense to use integer values when expressing lenghts in actual pixels, but percentages are real numbers anyways; even if you input an integer percent, the rendering engine calculates the final value in pixels as:
round(total * percent/100.0)
It's rounded anyways because most times "total" is not a multiple of 100, so there is literally no advantage to using integer values.
I tell you in case you thought it was better, maybe you are just that bad at math that "3 * 33 = 100"
>>20054
>current year
>not https
Step it up desu senpai:
https://letsencrypt.org/
I like it though, good job.
>>20054
I like it. First time coming to this board, glad I caught it.
>>20574
Nice association. The logo could then feature a space probe instead of a spider to mess with people.
Theme filtering is currently done offline. User-defined themes would require server-side code that would recognize the user or Javascript that would probably be too computing-intensive. At the moment I'm not convinced that many users would want to spend time to create logins or otherwise tune the experience. The generation of international themes could be done automatically in the offline code, though.
By the way, the layout is bugged in the theme view because the boxes are assigned to columns in the offline code and I haven't been able to get it around with the current SQL db. I know it could be done with MySQL but it just requires some effort.
>>20586
I think Bing translator is also a paid API. Now that you mentioned Twitter, I think I should next focus on tuning the share function for better Twitter support.
>>20630
I've always thought that using endless decimals is lame, and I couldn't use fractions so I left it to 33. But you are right, and the end-user doesn't anyway see the lame decimals but asymmetry instead. I fixed that.
>>20636
I'm not sure how SPIDR would benefit from https as long as it doesn't use server-side user identification. I honestly don't know, I will do that if you give me good reasons.
>>20653
>using endless decimals is lame
it is. I'm forever mad I cannot use arithmetic expressions in css, but after all its not a *real* language. Use 33.33% if it makes you happy, it won't change a pixel until the screen resolution is >15000.
>Theme filtering
How about a 4chan catalog-like search box above "Themes"? Not in real time, of course, I mean the algorithm.
http://boards.4chan.org/news/internet
I'd say you search the excerpts too, maybe add a "Title only" checkbox next to it. And I said right above "Themes" because if one day the themes overflow into many lines the search box would look awkward down there, it's best to have it right next to the word "Themes"
I don't enjoy bossing you around, but this is a very good service, take it as love senpai
>>20704
I was thinking of a simple text filter function to hide boxes that doesn't contain the keyword. That should be feasible to perform realtime with javascript. International themes are quite different as they require translation of the keywords.
Today I resolved the layout issue in the theme view, it was easier than I thought. I also added flags to indicate the source of origin.
Progression towards the mobile version can be found in http://spidr.duckdns.org/m/
Currently it looks quite crap but at least it has one column. I spent massive amount of time to make the layout more responsive, hope I didn't broke anything on the way. I will continue on that later.
>>20818
Good work, Anon. You should maybe post under a trip so we know it's you, the new King of /news/.
>>20822
Thanks, but I don't feels that's necessary.
Today I improved the mobile experience. Mobile phones are now redirected to the mobile version, I'm not sure should I first give a pop-up for the user to choose. There's a nasty bug in the expand box in some devices that when touched redirects the touch to the underlying link.
>>20983
perfect.
the last one, I swear:
I think the titles are too large. 150% or 170% instead of 200%? They are supposed to be easy to read, not jump in your face
>>21231
this
I like 150% best.
The "+20" button now works with mouseover.
Am I the only one that finds that invasive? I'm an app dev, and my experience is that the stablished languages is that mouseover is for flying stuff like tooltips and previews, but it should never restructure the general layout, it's not expected.
What triggers the whole page to move around should be something physical and that you choose to do, like a click.
The other day I saw a news disappear from the "North Korean trash baloon" group, that was only 4 hours old. Did you remove a news source? I thought they disappeared after 24 hours.
>>21333
just an fyi; I have checked the fact that the greatest part of you post number are consecutive numerals. Have a nice day.
>>21369
wait, no, forget the disappearing news, I dun goofed
>>20054
>>20128
Op I ran the site through a validator for css and html and they found some errors you should try to correct.
It works now probably but to ensure that less problems occur in the future you should try to fix the problems.
>css
jigsaw w3 org/css-validator/
>html
html5 validator nu/
put a dot where the spaces are
>>21369
My initial intention was to make it a button but then I noted that hover-action is also good. After all, the user isn't able to press the button without hovering above it and the opposite: if the user is hovering above, the next thing he will do is to press the button. So to save time, the action is triggered by hovering. I think it's unlikely for the user to accidentally trigger the action. In addition, the element doesn't look like a button that needs to be pressed, I should only find a way to emphasize that. But you are right in that this is contrary to common design principles. Anything is possible, I can make it clickable if users want that.
News might disappear because of two reasons:
1) The source feed has exceeded its capacity and the oldest news are thrown away to make room for newer ones. Typically feed sources contain about 20 items so it is very possible that high-activity feeds do not contain news older than few hours. I have plans to buffer the feeds so that the articles used by the offline code are retained regardless of the source feed capacity.
2) The group in which the news article belonged to doesn't exist anymore or was mutated in such a way that it doesn't match the disappeared article anymore. The aggregator discards articles that do not belong to any group because such articles are deemed as unique and thus likely not important.
>>21372
Thank you, sir. I will run the validators and do the required corrections.
Today I added filtering function. Excerpts are not included because it would have been too complicated. Otherwise it's quite nice.
>>21476
>to save time
I know where this come from, this way of thinking was beating out of me at work
>if the user is hovering above, the next thing he will do is to press the button
>I think it's unlikely for the user to accidentally trigger the action
no.
but maybe it's just my mouse hand's hyperactivity.
And no, they don't look like a buttons by themselves, but anything with a "+" universally conveys "something magical is going to happen if you click me", more so if it's a box hanging at the bottom of a list.
Either way I no longer care, wait for the next autist with a strong opinion about it.
In another, way more relevant matter, I cannot make the Filter work. I cannot even make sense of it, as I type letters whole columns dissappear, and never reappear when I clear the text.
also this error, idk if it's relevant:
>Refused to execute script from 'http://github.com/bartaz/sandbox.js/raw/master/jquery.highlight.js' because its MIME type ('text/plain') is not executable, and strict MIME type checking is enabled.
Just wanted to drop by and show support.
I love the name too, it's simple and relevant.
And the light-weightness is amazingly refreshing. I fucking hate web 3.0 bollocks. I work as a software tester for a company that is redeveloping their huge browser-only app to be all new and pretty, and it's extremely resource taxing and clunky. So don't add videos or embedded anything really, the site is beautiful as is.
tl;dr
looks interesting, but >ddns
>>21489
Okay, I promise to keep in mind the button function. The filtering effect had issues with Chrome but should now function correctly.
>>21502
Thanks for the support.
>>21506
Dynamic DNS certainly scares some people. I'm planning to register a domain if this thing escalates further.
Recently, I implemented the aggregator buffering function and added a bunch of german sources.
GET A TRIP
KING OF /news/
Also, I like the name.
>>20054
I'm viewing on mobile.
When I click the "+#" button to see more sources it's also clicking through the button and loading the source who's link is directing behind that button
I hadn't experienced that issue before today, though.
>>21935
Tripcode now in action.
>>21956
Yeah, that was reported earlier. You must have had good luck if this was the first time you encountered that. But now it is should be fixed.
I also made the +#-button clickable as insisted by >>21489. There was just not enough good reasons to keep the action hover-triggered.
upvoted
>>21976
Very true. Works much better now.you're a badass, btw
>>21976
just wanted to let you know that this has officially made it into my top five news sites to check in the morning
Well done, OPtitties
Have you thought about making a proper icon for it yet?
I just have a generic white page looking icon ;_;
>>22102
What are your others? I'm about to sound retarded, but I'm 30 and have no idea where to look for news...I only recently stopped watching TV news
>>22104
gonna catch some flak for this, I know it,
google news just for general aggregator
dailybeast.com probably second most objective site I use
rawstory.com, third objective and beginning to reach the gonzo-ne
alternet,org, not objective at all, but I like to see the perspective they present
upi.com is pretty well balanced and I like their odd news column
and I like techdirt.com for industry gossip, I find industry news is more onjective than bare market news
I stay away from all the major sites and political dividers like fox, msnbc, cnnstraight soap opera shit
>>22104
oh also, scientificamerican.com
theweek.com
and warisboring.com
>>20054bump
>>22209
This thread can't be bumped anymore, as it is now older than 48 hours.
>>22228
well alright then
The site has become slow to load because the layout is rendered column-wise. I've been struggling to modify the direction to row-wise so that it would at least appear to load faster but yet I haven't found a decent solution. Also, more things could be shifted to be done in client-side to reduce the download size.
>>22102
This was very nice to hear.
>>22103
There should be a grey S character in transparent background. What browser are you using? It could be that I messed up the favicon size.
>>22369
>There should be a grey S
Not that guy but I see the grey S fine, looks good. I'm using firefox on a PC though.
>>22369
that guy, grey S looks fantastic, using chrome on a chromebook
Now I managed to cut half the size of the main page by loading the hover content asynchronously. This increases the load on the server but reduces data traffic. I don't know which one is better but 1 megabyte of HTML was not acceptable. Mobile version is not affected by this.
>>22370
>>22378
Good to hear that.
>>22369
>a grey S character in transparent background
At the moment, in an unfocused tab of a focused firefox window, the grey color is a little too similar to the default grey of active firefox windows.
See screenshot: http://i.imgur.com/UOhOpoe.png
I suggest adding any kind of shadow/outline/detail that will help distinguish the shape more from similar greys. I made some quick mockup suggestions in the pic, and how they look in firefox and chrome. I prefer the middle one myself. I can't speak for other browsers right now, but with adding something like this, it should theoretically work in any browser regardless of color schemes.
It's a tiny tiny detail, but it'd at least make it a lot nicer for people using firefox.
>>22388
You know what, slightly disregard everything I posted - for a second I had the brainfart of thinking that my win8 color scheme was anything but that: just a personal win 8 color scheme that transfers to active windows too.
That being said, my suggestion is still valid enough, for the corner cases of people like me with similar greys. In general, having a single flat color for anything with a transparent background is not recommended.
>>22412
i like it
I cut the time threshold from 24h to 18h. Also, groups with only two news are now discarded. This trick reduced the size further to about 300 kB. I believe the rendering is now fast enough without any special tricks.
Only minor improvements are yet to be done so I think it's soon the time to shift up. I registered a domain with no bandwidth limit for this purpose at http://spidr.today/
>>22663
That's a really elegant site name now. I like it.
>>22663
well done
wondering if there is a way to efficiently create a tag system, something that could narrow the categories of the available articles
also wondering if any sporting news will make the cut, I've noticed a scant few articles on criminal related matters, but not much on sport in general
is this a subject that you would rather avoid or separate from the general feed?
>>22663
i only have four icons on my main bookmark bar, everything else in a folder system, and SPIDR is one of those on my main bar
flawless execution, keep up the good work
>don't go wearing yourself out dough -_-
>>22701
Well described, sir.
>>22744
In my opinion, sport news would not benefit from the SPIDR idea because they are way less colored than political news, for example. No source would claim that 'the Panthers actually won the Superbowl'. In contrast, many sources say that Donald Trump has good thoughts and many sources claim that he's a bad guy. I don't personally follow sport news that much, and I don't know how much the contents of the same topics vary between sources but I have the feeling that there are not many differences.
However, I am excited about the Rio olympics. It is highly international theme and probably will bring us not only sport results but also scandals, crime, politics and so on. I might well reconsider adding sport-categorized feeds then, we'll see.
In general, it would be easy but exhausting to add higher-level categories. Many sources already categorize their feeds to politics, world, sports, entertainment, economics and so on. I would just need to go through all the sources and map the category to feeds in the offline code.
>>22747
I'm very pleased to hear that.
New thread when this dies, SPIDR? Also, could you add weather that you can select the area of to the home page, if anyone else wants it?
This weekend I've been mostly improving caching so that the server would not have to execute scripts so often. Some work will still need to be done in the following week. I also made measures in the offline code to prevent over-classification.
>>23101
Hmm, I think there are plenty of weather sites out there and building such service is beyond my interests. I would prefer to keep SPIDR solely for news. Although it is a very nice idea to have a weather aggregator that would display the forecasts of many sources because sometimes there are differences between them. I don't know if such services already exist.
We're on now page 9. I don't know when this thread will be killed but I will be following this board in the future as well.
>>23178
Do you mean the SPIDR address? The official address is http://spidr.today/
>>23181
Yes, if only because a) I'm too lazy to use bookmarks and b) this is pretty useful and newcomers to the board won't necessarily have seen this thread once it 404's
>>23181
If you find something for weather, I think that it'd be pretty rad to have that.
>>23322
weather and sports agg extensions are a dime a dozen
like he said, it isn't really worth his time
SPIDR is supposed to be focused on current news reports
...........@@@@@@@@@@@@@@@@@@@@@@@
.......@@@@@@@@@@@@@@@@@@@@@@@@@
....@@@@@@@@@@@@@@@@@@@@@@@@@@@
.@@@@@@@@/███████`#@@@@@@@@@@@@@@@
@@@@@@@,;█████████$@@@@@@@@@@@@@@@
.@@@@@@/████████████%*#@@@@@@@@@@@@@
..@@@@,./█▀████▄███████,%@@@@@@@@@@@@@@
....@@@,/██████████████.*&*%@@@@@@@@@@@@@
......@@/████████████████g$@@@@@@@@@@@@@@
.......@/█░░░░░░░██████████`%@@@@@@@@@@@@
........,/█░██████░█████████&@@@@@@@@@@@@
......./█░██████░██████████`%@@@@@@@@@@
.....,/█░██████░███████████#$@@@##@@@@/
...,/█░░░░░░░░███████████████;`=””=.`;@#*
.,/████████████████████████████_*”`
(███████████████████████████;`
.`\,█████████████████████████`
.......`-,████████████████████/
.............\██████████████████|
THREAD CLOSED
posting in an historic /news/ thread before 404
thank you again SPIDRanon for all you have done
>>23627
We got this screencapped?
Shouldn't we come up with a nickname for the guy? We are not going to call him Spidrman forever
What does Spidr even mean?
>>22830
>reconsider adding sport-categorized feeds then
>>23181
>I would prefer to keep SPIDR solely for news
you sound like you want to add a "2016 Olympics" one, not "Sports". I agree sports might not benefit much from the format, and the site works best for news. I can't make my mind about a "Sci & Tech" tag though, so I'm just throwing it out there. Because there are sometimes are different views, like about news on human genetic trials and such.
Do groups support more than one tag? As in "Business" and "Sci & Tech" for things like an eventual Google+SpaceX shared project?
Also, it's been a while and thanks for the little things, it loads better and very nice domain. I'm the "CSS and button functionality" guy, so here's one last autism:
add "cursor: pointer;" to the block ".expb" to better convey it's a button (and so you don't see the "select text" cursor over the "+#").
My friends like the page too, so good job Spi.
>>23706
>We are not going to call him Spidrman forever
Why not
>What does Spidr even mean?
There is already a thing called a spider, that some search engines use to gather results. If you've ever been on phpbb forums, you will have noticed different search engines having spiders present on the online user list. I assume this is what he thought of when choosing the name, and if not it still makes sense as a crawly thing that gathers stuff in a web.
http://newsmap.jp/
Oh, hi
I forgot about this thing
it's a Google based aggregator that has been in beta for half a decade, nothing really changes about it much but the articles
Enjoy
>does this board even have a moderator? =\
>sticky dammit!!
