Update on ReGENESIS: API and Slides

24 August 2013

It’s been two weeks since I’ve first blogged about ReGENESIS, and the project has gathered some very positive feedback. I’ve been able to present the site to a number of people and to extend the functions of the site. Here’s a few quick updates:

JSON API for statistics

One of the key value propositions of ReGENESIS had been the option to get live statistical data and to use it to quickly render out visualizations and charts. This means providing functions like filters, aggregations and web data formats like JSON.

I’d initially been relucant to run an API server for ReGENESIS, but I’ve now come to believe that creating a purely flat-file driven interface is not possible. That’s why I’ve now extended the package to generate configurations for Stefan Urbanek’s excellent Cubes package.

This service provides a simple, OLAP-inspired HTTP API which can be used to easily access each of the data cubes provided by the Regionalstatistik portal. Check out the ReGENESIS API introduction.

Hacks/Hackers in Berlin

Thanks to Annabel, I was able to present ReGENESIS at this week’s Hacks/Hackers meetup in Berlin to an audience of tech and journalism enthusiasts. The pitch was aimed to explain the advantages of ReGENESIS data format to non-experts, and to call them to try out the data provided by the service.

I’ve embedded the slide deck below.

Full set of statistics

Soon after I’d published the initial post about ReGENESIS, people started pointing out that some of the statistics published on the Regionalstatistik site were missing from the database. This was caused by a bug in the initial scraper, which I’ve now fixed. This means the number of datasets published on the site has near doubled, including the availability of electoral statistics for many of the German federal states.

Other blog posts

  • A little tour of aleph, a data search tool for reporters

    Over the past six months, I've been working for OCCRP to productise Aleph, a powerful search tool for investigative reporters. This is a little tour of it's key features, and a little view into the future development agenda.

  • A Poor Journalists's Text Mining Toolkit

    How can journalists search and analyze collections of documents on their own computers with simple tools? At last weekend's DataHarvest, we ran a workshop trying to answer that question. This write-up to covers using Apache Tika for content extraction and regular expressions in Sublime Text as an advanced search tool.

  • Against Decentralization

    In the free software/open web community, the notion that the web should be decentralized is more than a shared ideal, it is a piece of dogma. But are we really promoting a progressive vision of the web, or fighting a losing battle to avoid political engagement?

  • Keeping stock: investigative data warehouses

    Data warehouses are used in industry to manage the many datasets accrued inside a company that might be relevant to reporting and analysis. I want to propose a similar pattern for investigative journalism.

  • SpenDB, a data analysis tool for government finance, looking for testers!

    The first beta version of SpenDB features a small set of well-designed features for data import and analysis. Now the platform is ready to be adopted by anyone interested in exploring financial data, from budgets to procurement.

  • On Hacks/Hackers, Google and community building

    A few weeks ago, the US team of Hacks/Hackers announced their plans to turn the network of journalism innovators into a collaboration with Google News Labs, starting with an event in Berlin. I tweeted about this, and Phillip Smith wrote a thoughtful reaction. Given this invitation to debate, I wanted to outline my criticism in more detail.

  • SpenDB, a light-weight tool for government financial data

    Over the past few months, I have spent my weekends simplifying and modernizing the OpenSpending codebase to create SpenDB - a prototype-stage, light-weight data loading tool and analytical API for government financial data.

  • Who’s got dirt? - What if robots could do cross-border investigations?

    If we want to make open data relevant to investigative journalism, we have to simplify the way people access it. We must create a way for our data tools to talk to each other and trade information about the companies and people we are researching.