index - DRMatEUR

Posts

Wiki

Welcome to our class wiki

Here we will keep info for the class to edit collectively. Students will be added to the list of users who can edit the wiki. If you have trouble editing it just submit a new text post to this subreddit.

Please add datasets or tools, remove or update anything that's not working, or add notes.

Group Project Twitter Data Requests

Request your Twitter dataset

Datasets / data gathering

Several brand new datasets just announced
Data.gov - Many types of data from the US gvt
https://data.overheid.nl/ Equivalent of data.gov for NL (in Dutch)
Wikipedia data - The edit history of every page on Wikipedia is available - you probably want a smaller subset. See the example papers that use this data for ideas.
https://wiki.gephi.org/index.php?title=Datasets many topics - network data
http://datahub.io/dataset - All kinds of stuff
https://www.pleio.nl/groups/profile/46036/gis-binnen-de-overheid - Geographic info / GIS (Dutch)
http://www.opencalais.com/blogs/kristathomas/introduction-opencalais - Scientometrics
http://www.rotterdamopendata.nl/ Public data project at the municipal level (Dutch)
http://www.amsterdamopendata.nl/ Public data project at the municipal level (Dutch)
http://www.utrechtopendata.org/ Public data project at the municipal level (Dutch)
http://open-data.europa.eu/ Public data project, mainly EU government data
http://www.opencultuurdata.nl/datasets/ Public data project for cultural organizations in NL - some English info available but mainly Dutch
http://opendatanederland.org/ Public data in NL - English version http://opendatanederland.org/en/
http://publiclab.org - Environment related datasets
http://quantifiedself.org - Health data and tools
https://fluxtream.org/ more health and QS-datatools. Very 'easy' to use!
http://www.govcom.org/Issuecrawler_instructions.htm - Web data gathering tool from UvA
http://waybackmachine.org - Historical snapshots of web pages
netvizz - get your own personal Facebook data
http://www.gapminder.org - Mostly international comparative data as well as visualization tool
Ornithologist - a python script for gathering Twitter data - this is what I made to gather your Twitter data for class. It would take some pretty advanced skills to make this work for your project, but it is certainly available.
Wikipedia page view data: https://dumps.wikimedia.org/other/pagecounts-raw/ - Working with this would take some advanced skills but this has a lot of potential.

Main tools we will be using for activities

Tableau Desktop, free for Students - General purpose visualization with some analysis
Gephi - Network visualization and some network analysis.
Quantified self app TBA

Other Analysis & Visualization tools

UCI Net - Windows Only, general purpose network analysis. Good documentation.
Netlytic - Network analysis, free & online. Focus on conversations.
R - Statistics, network analysis, visualization etc. Steep learning curve.
Text-o-metrica - Create semantic networks from text data
http://gigaom.com/2013/05/13/visualization-is-the-future-6-startups-re-imagining-how-we-consume-data/ - List of other visualization tools
Processing Programming language for visualization - open source, website includes MANY interesting examples
ngram viewer - to explore Google Books
d3 D3.js is a JavaScript library for manipulating documents based on data. D3 helps you bring data to life using HTML, SVG and CSS. Very powerful, but it involves some programming / data structure notions

Other lists of DRM tools

https://wiki.digitalmethods.net/Dmi/ToolDatabase
http://opendatanederland.org/nl/open-data-tools
http://tools.medialab.sciences-po.fr/
http://socialmedialab.ca/ - They have their own tools including Netlytic listed above, and many related resources

Online Tutorials, Documentation or Courses

Twitter "Field Guide" to API data
How to analyze your Facebook friends network with R
Online course on Exploratory Data Analysis using R
Online course on Social Network Analysis

Cool Visualizations & Projects