SpenDB, a data analysis tool for government finance, looking for testers!

30 June 2015

Earlier this month, I wrote about the prototype of SpenDB, a light-weight analytical tool for government financial data. After nearly a month of work, the tool has now matured into a beta release, ready for thorough testing with your data!

SpenDB data loader
Checking out the largest suppliers of World Bank projects in South Africa.

The two most obvious changes are the re-designed user interface with a much simpler and quicker data import workflow, as well as the inclusion of angular-cubes, a simple query tool that can be used to browse data in pivot tables and treemaps. I’m hoping to add further visualisation types, like bar charts, line graphs or sankey diagrams in the near future.

This makes it easy to upload a CSV or Excel file, validate its contents, model the data and then run some rudimentary analysis within minutes. It also means the API to drive third-party applications is now well-tested and stable, as today it will begin to be used by OffenerHaushalt as it’s backend.

Unlike OpenSpending, I don’t expect SpenDB to be used only as a hosted service. The tool is easy to install and can be customized and used for parts. If you’re interested working with the project - as a user or contributor, give me a shout or join the Gitter chat.

Finally, for the more technically inclined, a more thorough list of major changes:

  • Switched the frontend from server-side templates to a stand-along Angular application. This is beginning to turn into a stand-alone package, thanks to Mila.
  • Improved data validation and source data analysis, compliant with JSON Table Schema.
  • Built and included angular-cubes to run simple data anlaysis on each fully loaded dataset.
  • Direct-to-S3 uploads from the user interface to allow for stable uploads of up to 5G source data.
  • Split out dataset reference data into a separate package, fiscalmodel.
  • Simplified the asset builder and the client-side caching subsystem.

With all of this infrastructure work done, I’m hoping for the next steps of this development to relate more closely to specific datasets. This includes extending the functionality of OffenerHaushalt, but also making more granular datasets, like the Förderkatalog accessible.

Finally, I want to tackle the big one: OpenTED, which currently exists mainly as a collection of CSV files, will hopefully gain an accessible query interface so that journalists and researchers can more easily approach it.

Obviously, I’d love to learn more about how other users would want to see the tool evolve, and to accept any pull requests and changes!

Other blog posts

  • A little tour of aleph, a data search tool for reporters

    Over the past six months, I've been working for OCCRP to productise Aleph, a powerful search tool for investigative reporters. This is a little tour of it's key features, and a little view into the future development agenda.

  • A Poor Journalists's Text Mining Toolkit

    How can journalists search and analyze collections of documents on their own computers with simple tools? At last weekend's DataHarvest, we ran a workshop trying to answer that question. This write-up to covers using Apache Tika for content extraction and regular expressions in Sublime Text as an advanced search tool.

  • Against Decentralization

    In the free software/open web community, the notion that the web should be decentralized is more than a shared ideal, it is a piece of dogma. But are we really promoting a progressive vision of the web, or fighting a losing battle to avoid political engagement?

  • Keeping stock: investigative data warehouses

    Data warehouses are used in industry to manage the many datasets accrued inside a company that might be relevant to reporting and analysis. I want to propose a similar pattern for investigative journalism.

  • On Hacks/Hackers, Google and community building

    A few weeks ago, the US team of Hacks/Hackers announced their plans to turn the network of journalism innovators into a collaboration with Google News Labs, starting with an event in Berlin. I tweeted about this, and Phillip Smith wrote a thoughtful reaction. Given this invitation to debate, I wanted to outline my criticism in more detail.

  • SpenDB, a light-weight tool for government financial data

    Over the past few months, I have spent my weekends simplifying and modernizing the OpenSpending codebase to create SpenDB - a prototype-stage, light-weight data loading tool and analytical API for government financial data.

  • Who’s got dirt? - What if robots could do cross-border investigations?

    If we want to make open data relevant to investigative journalism, we have to simplify the way people access it. We must create a way for our data tools to talk to each other and trade information about the companies and people we are researching.