A good social network analysis tool for advocacy & journalism

19 December 2011

Some of the most interesting transparency and journalism-related datasets are those in which the relations of various actors and institutions are portrayed over time: corporate ownership, lobbying or organized crime networks all present similar challenges.

And yet there seems to be no good off-the-shelf package to display such data. Sure, there are things like UCINet for classic SNA and Gephi for graph data visualization. But I think they’re inadequate. The gold standard for graph presentation in 2011 is not graphviz, its Facebook. I don’t mean to imply that networks need to be stored as massively centralized, ad-powered dotcoms, but there are a couple of lessons to learn:

  • It needs to be fully web-based and not overtly about graph management. Rich attributes for both nodes and edges need to be stored and presented in a task-specific form.
  • Some level of access control enables more interesting use cases.
  • Network effects apply to networks. Ha.

The problem in building this is simple: its a trap. For any coder, our greatest dream is to transfer some new domain of knowledge into a perfect graph (I’m guilty myself, having worked on liquid democracy). In this case, the temptation is almost irresistible: the data is already a network.

Yet as we work to translate things into this graph, we tend to lose sight of the goal. A nice example of this is OWNI’s InfluenceNetworks: while it is supposed to be the the tool this whole post is about, they ended up just building a snazzy graph management app. Nice, but useless.

Even more dangerous: doing this internally. The linked data community seems to be crippled by graph fever. More than once have I sat through an hour-long presentation on the use of linked data in which no actual use for an end-user facing application was mentioned. While I’m not saying its impossible to build good applications on RDF, it seems like a huge distraction from actually tackling the domain.

But the problem remains: we need some easy-to-use solutions for advocates, journalists and other investigators to map out networks. The best effort towards this I’ve seen so far seems to be andoc, but even this does not feel like it is hitting the right trade-off between abstract graphness and concrete modelling yet. But: how do we find that trade-off? Seems some conrecte use cases and a nice agile process is needed…

Other blog posts

  • A little tour of aleph, a data search tool for reporters

    Over the past six months, I've been working for OCCRP to productise Aleph, a powerful search tool for investigative reporters. This is a little tour of it's key features, and a little view into the future development agenda.

  • A Poor Journalists's Text Mining Toolkit

    How can journalists search and analyze collections of documents on their own computers with simple tools? At last weekend's DataHarvest, we ran a workshop trying to answer that question. This write-up to covers using Apache Tika for content extraction and regular expressions in Sublime Text as an advanced search tool.

  • Against Decentralization

    In the free software/open web community, the notion that the web should be decentralized is more than a shared ideal, it is a piece of dogma. But are we really promoting a progressive vision of the web, or fighting a losing battle to avoid political engagement?

  • Keeping stock: investigative data warehouses

    Data warehouses are used in industry to manage the many datasets accrued inside a company that might be relevant to reporting and analysis. I want to propose a similar pattern for investigative journalism.

  • SpenDB, a data analysis tool for government finance, looking for testers!

    The first beta version of SpenDB features a small set of well-designed features for data import and analysis. Now the platform is ready to be adopted by anyone interested in exploring financial data, from budgets to procurement.

  • On Hacks/Hackers, Google and community building

    A few weeks ago, the US team of Hacks/Hackers announced their plans to turn the network of journalism innovators into a collaboration with Google News Labs, starting with an event in Berlin. I tweeted about this, and Phillip Smith wrote a thoughtful reaction. Given this invitation to debate, I wanted to outline my criticism in more detail.

  • SpenDB, a light-weight tool for government financial data

    Over the past few months, I have spent my weekends simplifying and modernizing the OpenSpending codebase to create SpenDB - a prototype-stage, light-weight data loading tool and analytical API for government financial data.

  • Who’s got dirt? - What if robots could do cross-border investigations?

    If we want to make open data relevant to investigative journalism, we have to simplify the way people access it. We must create a way for our data tools to talk to each other and trade information about the companies and people we are researching.