Why I'm building Grano, and why you should help

30 March 2014

Over the past weeks, I’ve spent a lot of time working on grano, a social network analysis toolkit for news applications. My goal is to make a reusable, adaptable and open source tool that helps journalists and researchers to map relations between people, companies and institutions.

BP on OpenInterests.eu
OpenInterests.eu is a data application based on grano.

While there are many awesome projects working along similar lines, none of the platforms have found wide adoption. Yet sharing technology is essential, as the effort required to make network analysis useful for understanding social systems is substantial.

The domains in which people use social network analysis are diverse, from lobbying and political corruption and company ownership to music bands or fictional characters. That is why grano isn’t a ready-made platform to model all of society. I want to avoid converging on the lowest common denominator of power mapping: drawing busy network diagrams which may be pretty, but are often poor tools for analysis.

Rather, the goal is to build a toolkit of APIs and task-specific components that can be assembled to fit a particular task and to support whatever approach is needed to understand the domain at hand. My central goal is to make grano an open source project; not just in terms of it’s license, but by creating the kind of documentation and installation process that makes it easy for people to contribute and build upon the existing code.

Grano
The prototype of the grano administration interface allows users to manage and explore entities and their relationship.s

The database at the foundation of grano is rich with metadata, allowing users to define the types of entities and relations that their project requires; who has access to the data; and to store the source and history of each fact stored by the system. To accomodate this information, the Flask-based Python application runs against a Postgres database as its main store. At the same time, I’m planning to support Neo4J as a secondary store to be able to perform powerful graph queries for those use cases which require them.

At this point, grano is already used to run OpenInterests.eu and Kompromatron. Both applications combine large sets of raw data from a variety of sources that refer to the same set of entities (EU organisations in one case, German politicians in the other). This type of data integration - combining many sets of relationships into a combined network - is a scenario that I want to focus on. That means building the means for non-technical users to perform raw data imports and entity deduplication through grano administration interface.

So, this post is a call for contributors: people who want to contribute to the application; but also those who would like to analyze power structures by means of mapping them as a network. Please get in touch, file issues on GitHub, send pull requests and follow @GranoProject on Twitter for updates on where this journey is going.

Other blog posts

  • A little tour of aleph, a data search tool for reporters

    Over the past six months, I've been working for OCCRP to productise Aleph, a powerful search tool for investigative reporters. This is a little tour of it's key features, and a little view into the future development agenda.

  • A Poor Journalists's Text Mining Toolkit

    How can journalists search and analyze collections of documents on their own computers with simple tools? At last weekend's DataHarvest, we ran a workshop trying to answer that question. This write-up to covers using Apache Tika for content extraction and regular expressions in Sublime Text as an advanced search tool.

  • Against Decentralization

    In the free software/open web community, the notion that the web should be decentralized is more than a shared ideal, it is a piece of dogma. But are we really promoting a progressive vision of the web, or fighting a losing battle to avoid political engagement?

  • Keeping stock: investigative data warehouses

    Data warehouses are used in industry to manage the many datasets accrued inside a company that might be relevant to reporting and analysis. I want to propose a similar pattern for investigative journalism.

  • SpenDB, a data analysis tool for government finance, looking for testers!

    The first beta version of SpenDB features a small set of well-designed features for data import and analysis. Now the platform is ready to be adopted by anyone interested in exploring financial data, from budgets to procurement.

  • On Hacks/Hackers, Google and community building

    A few weeks ago, the US team of Hacks/Hackers announced their plans to turn the network of journalism innovators into a collaboration with Google News Labs, starting with an event in Berlin. I tweeted about this, and Phillip Smith wrote a thoughtful reaction. Given this invitation to debate, I wanted to outline my criticism in more detail.

  • SpenDB, a light-weight tool for government financial data

    Over the past few months, I have spent my weekends simplifying and modernizing the OpenSpending codebase to create SpenDB - a prototype-stage, light-weight data loading tool and analytical API for government financial data.

  • Who’s got dirt? - What if robots could do cross-border investigations?

    If we want to make open data relevant to investigative journalism, we have to simplify the way people access it. We must create a way for our data tools to talk to each other and trade information about the companies and people we are researching.