A while back, I did some visualizations on Harry Potter fan fiction, scraped from FanFiction.net. To do this, I built a web scraper in Python, saved off the data as one huge CSV file, and had my visualizations pull from bits and pieces of this data file.
While this worked well for the data that I had, I found myself wishing that the data could be updated and that I could see in real time when fan fictions were being published. Even though it has been years since a Harry Potter book came out, in my scraping adventures, I found that people were still publishing their writing every hour or so! With the method I used, all of my data stopped on the day I happened to scrape it. How would I go about being able to continuously show updated data?Read
Update (1/24/2018): This is now deployed on Heroku!
For the past few weeks, I have been working on a project that I am currently calling Wikigraphs. What in the world is that, you ask? Well, thanks for asking! If I had to summarize it, I'd say it is a framework that allows for community driven data as input into data driven graphs, thereby allowing for transparent data sources.
Let's break that down.Read