The social media analysis tools already definitely exist, and there's no shortage of other anons who have a decent understanding of programming for apis as well as just making web apps, or even using existing tools.
So, wanna form an analysis group to see how different concepts (such as how well Trump's being received in battle ground states, or how many people are calling for him to be assassinated) are spreading around on social media such as facebook, reddit and twitter?
We wouldn't be working for any one group per se, just better informing the rest of the /pol/incos about the actual state of things outside the hugbox.
>No script kiddies plz
>>67121445
Neat, what can you do NEET?
>jJust a heads up, but everyone who comments is just a neet until proven otherwise.
>>67121566
I troll hard
>>67121868
While an important tool, it sounds like you don't actually know how to use the tools we'd be working with (like Iconosquare or Collecto)
(Also, what's with South Korea's negative population growth rate?)
afaik twitter shut down everyone's ability to tap into the firehose
or has this changed since they had their ipo and have been in freefall ever since?
>>67123065
If you know how to program in ruby you basically know how to use the twitter api and can make customized tools to see specific information (for example, how many followers Trump gains after events like debates and rallies).
So, things like their new "safety" commission are irrelevant to this, or are you referring to something else entirely?
>>67123697
he might be talking about how they rate limit requests. The last I looked (earlier this year) you have a limited number of requests per hour you can make
>>67121284
I'm interested. I have college so my free time is limited, but I'll see what I can do. I could make an anonymous github account and move stuff there.
>>67123697
im aware of how to manually scrape and diff sites
its way more convenient to be able to run nlp and other things without having to clean data up first
what does ruby have to do with anything? you sound like a faggot
also yeah talking bout rate limits and other restrictions. they locked their data down into proprietary mode like niggers
>>67124572
would a custom web scraper be a decent replacement?
>>67124163
I'm thinking that's what anon was getting at. Fortunately, there's numerous tools that can be used free of charge to spread out data acquisition, allowing us to use the limited requests (which are still pretty large) for things which can't otherwise be acquired.
Here's a few:
https://blog.bufferapp.com/social-media-analytics-tools
>>67124459
It's the language you would make requests using the Twitter API with most easily. There's others obviously, but almost all of the documentation for the Twitter API is specifically for Ruby. Feel free to prove me wrong tho.
You do anything particularly notable with your skill set btw?
>>67124572
Not all of it though, a lot of it is still readily accessible for fuckwits like us.
>>67124814
if you're determined enough yeah. i don't really know of any other viable options.
the thing is you'd have a lot of hoops to jump through to run any kind of in depth analytics as far as tying things together
i guess if all you want is whats being said then its not a big deal
im only talking about twitter btw.
>>67125233
good point. getting number of new followers would be fucking painful
>>67121284
stackechange crossvalidated
>>67125233
>>67125367
Not impossible though, and we'd simply be sorting through the available trove of data that can be culled from not just Twiiter, but also Facebook, Reddit, Instagram (lel), et al. Putting together graphs of new followers per event wouldn't really be necessary, if we can assemble data sets fully leveraging what we can get for free from the API while working cohesively.
(And yeah, the tools that let you really get into the thick of it require payments to the service which open up a greater portion of Twitter's datasets, wouldn't be a great investment imo)
>>67125033
i know twitter started out as a topic based pub sub ran off of mysql that couldn't scale and would have gone bankrupt/the way of the dinosaur if they didnt have $100 mill+ in capital at the time. i guess youre saying the functionality theyve exposed to the public is written in ruby? i'm not trying to pick a fight about it, i'm just not entirely certain why this matters since you'll be getting serialized data from their api endpoints anyway and that stuffs gonna be language agnostic
anything particularly notable? idk. i try to be humble when it comes to code because the more you learn the more you realize you don't know.
>>67126006
yeah. ive thought about doing this sort of stuff before. i was going to have a raspberry pi set up to scrape on certain intervals, throw everything into some sort of data persistence, and run diffs or whatever was appropriate to figure out what i wanted to know.
either way im a total junkie when it comes to comp sci / software engineering so i enjoy thinking about solving problems.
>>67126163
It's simply what their api that allows for (admittedly limited) access their accumulated data calls for, idk what else to really tell you.
Bump for potential, sounds pretty cool
contact me on twitter @ostpeppar
>>67126912
idk mang im kinda operating under the assumption were talking about a RESTful JSON api. if not then im fucking retarted and you can tell me so.
i would like to contribute to a project that strikes my interests and something in this area definitely would.
specifics on my skillset would be node, garden variety frontend js, frontend js frameworks/libraries, various frontend build tools, various databases (sql, nosql, graph), and other backend / server side languages. also aws and azure. i fuckin hate css/sass/stylus and all of that shit though and i only do it for money
>>67126163
>i try to be humble when it comes to code because the more you learn the more you realize you don't know.
autism
>>67127694
hey i heard theres another weapons sweep. go hide your butter knives.
I'm very interested- I have experience working with HUMINT tools like Maltego CaseFile, and some very interesting memetics frameworks (pic related) figured out that I have no one to really share with in a productive manner.
Discord is a good tool for groups like this.
https://discord.gg/tetsudo-priest-sahagin
>>67127546
Based on further reading from the developers site I'm actually pretty sure you're correct lel.
https://dev.twitter.com/overview/general/things-every-developer-should-know
That also sounds a bit further along than my own experience with front/back end web dev so you could very easily lead the way in this realm of data shenanigans.
I guess the only two questions I've got left are: have you ever participated in any data science competitions (like kaggle) and what would be a decent starting point for such an ambitious project, in your opinion?
>>67127821
kek
>>67128300
Discord may be great, but I'm all for keeping this completely anonymous
>>67128300
That's awesome actually, tracking the spread of ideas is critical for this. Do you have any experience working with things like hadoop (I imagine we'd prolly use Apache Giraph) for sorting through all the data we might end up assembling?
>>67129003
Well, you're going to need a data repository and some form of organization- simply doing this on a chan won't work.
I'd suggest at the very least a bitbucket + JIRA.
>>67128813
i've only dabbled in machine learning and data science. it has always been near the top of my list of things i plan on deep diving into but i haven't made it there yet.
i guess i'm more of a system architecture kind of person? dunno.
as far as starting point.. drilling down to a specific problem we want to solve at first. then i think reading through the api docs for the different social media platforms to see what functionality they expose to the public is a good next step. from there it would be easier to determine what we would have to implement ourselves in order to cross reference and do other things.
>>67129417
agree with the necessity of some form of dvcs at the very least
>>67129296
>>67128300
I have the technical knowledge to converse about and design data frameworks, but I am a sub-par programmer. I can read code though, I'm just not the most efficient when it comes to implementing code.
I've worked as a SQL analyst before for a major software company, but my background is in history, philosophy, economics, and science fiction.
I honestly believe that the problem of intelligent analytics is a matter of bad hardware design and wrong-thinking, not a matter of difficulty.
>>67129417
dang i didn't know you can self host JIRA for $10
that sounds like a ridiculously good deal.
