- Disorganized content: Existing social media generally create very disorganized content, so it's time-consuming to find what has been said on any topic of interest. This fosters unsystematic coverage, since users are often unable to quickly identify which areas aren't yet well-covered and need more attention.
- Low signal-to-noise ratio. Social media content is notorious for producing highly redundant content, so enormous effort is typically required to "harvest" this wisdom to inform better, more broadly-supported decisions.
- Quantity rather than Depth. Social media systems often elicit many relatively small contributions, rather than a smaller number of more deeply-considered ideas, because collaborative refinement is not inherently supported.
- Polarization: Users of social media systems often self-assemble into groups that share the same opinions, so they see only a subset of the issues, ideas, and arguments potentially relevant to a problem. People thus tend to take on more extreme, but not more broadly informed, versions of the opinions they already had.
Here's just two examples (see this blog post for more): (1) Intel ran a web forum on organizational health that elicited 1000 posts from 300 participants. A post-discussion analysis team invested over 160 person-hours to summarize these contributions (at 10 minutes a post, probably longer than it took to write many of them in the first place). The team found that there was lots of redundancy, little genuine debate, and few actionable ideas, so that in the end many of the ideas they reported came from the analysis team members themselves, rather than the forum. (2) Google used their "moderator" system to collect ideas for which charitable projects the company should fund (project10tothe100). The company had to recruit 3,000 employees to filter and consolidate the 150,000 ideas they received in a process that put them 9 months behind their original schedule. The vast majority of these ideas were minor variants of simple suggestions (e.g. support public transport, make government more transparent, and so on). Surely that vast amount of effort could have been used to compose a smaller number of more deeply-considered ideas, rather than many shallow ones.
A screenshot from the Deliberatorium. Each line in the left pane represents a single issue, idea, or argument. Each such post can have it's contents viewed, edited, discussed, and rated in the right pane.
The community follows a simple process to ensure that the deliberation map is as useful as possible:
Every author starts by "unbundle-ing" their contributions into individual issues, ideas, and arguments, and adding their unique points to the relevant part of the map. A key tenet is the "live and let live" rule: if you disagree with an idea or argument, you shouldn't change the post to undermine it, but should rather create new posts that present the strongest ideas and (counter-)arguments you can muster, so all contributions can compete on an even basis.
Moderators help ensure that these guidelines are followed. New posts can, at first, only be viewed by moderators. When a moderator verifies that a post follows the deliberation map guidelines, they can be viewed, edited, commented, and rated by the full community. If a post doesn’t yet meet the guidelines, the moderator leaves comments explaining what needs to be done to fix them. Moderators play a modest “honest broker” role: their job is not to evaluate or change the content of a post, but simply to help authors ensure that the content is as accessible as possible to the community at large. Moderators can check each other's work, as they do in such systems as Wikipedia and Slashdot, to ensure they are doing a good job.This process is supported by such software capabilities as open editing (any user can check and improve posts), watchlists (which automatically notify users of changes to posts they have registered interest in) and version histories (to allow users to roll-back a post to a previous version if it has been "damaged" by an edit). The system also provides a powerful set of attention allocation metrics to assess how well each part of the deliberation is going, and thereby to help community members focus their efforts where they can do the most good.
The current user interface, while functional, obviously has an "old world" feel. This reflects my limitations as a programmer, as well as a strategic calculation. I've noticed that many hugely successful web 2.0 systems (such as email, news groups, Wikipedia, and Facebook) became that way despite having initially quite rudimentary user interfaces. The key success factor often seems to be the compelling-ness of the idea, rather than the polish of the implementation. I've focused, therefore, on rapidly exploring new ideas, rather than polishing old ones. Going forward, my hope is that the most powerful ideas explored in this work will be taken up by commercial vendors and incorporated in their products.
The carbon offsetting web forum summary map
This was a startling illustration of the potential of deliberation maps for harvesting a community's collective knowledge in a way that is qualitatively more useful than conventional social media.
Our first large-scale evaluation was at the University of Naples, where 220 masters students in the information engineering program were asked to weigh in, over a period of three weeks, on what use Italy should make of bio-fuels. All told, the students contributed nearly 2000 posts, creating a map that was judged by content experts to represent a remarkably comprehensive and well-organized review of the key issues and options around bio-fuel adoption, exploring everything from technology and policy issues to environmental, economic and socio-political impacts.
A small portion of the Naples bio-fuel deliberation map
We found that, initially, about 2/3rds of user posts were structured correctly as originally created, and this increased to about 85% by the end of the deliberation. Also, the remaining posts almost always just required some simple fix e.g. people would call it an "idea" when it should have been a "pro" (see this paper for details). Overall, we were encouraged by how well people could use the deliberation map structure. It made it possible to support the user community with the part-time support of just two moderators. We were hard-pressed to imagine any other approach that would allow over 200 authors to write what was in effect a substantial book on a complex subject, in a couple weeks, with no one in charge.
Our first business-centric evaluation was conducted with Intel Corporation on the question of how "open computing" (i.e. where users are given greater access to computing tools and data) should be used in the company. Contributions were purely voluntary. A single moderator was able to support the discussion with very little effort. The end result (see below) was that Intel received a substantive and well-organized overview of important issues in this space from 73 contributors, including many from outside the company, at close to zero cost.
Map generated for the Intel deliberation on open computing
We've also conducted evaluations with the US Bureau of Land Management and the University of Zurich, among others, and we've learned that, when compared to conventional social media, it can help you get better content on how to solve complex problems, at lower cost, small voices can be heard and, perhaps most importantly, it's much easier to find the "good stuff".
1. Embed the tool into a management process so that it's part of the "workflow" and taken seriously (for instance, make it a part of the strategic planning process). Your organization's members are much more likely to try a new approach if it is part of a conversation that matters.
2. As an initial test of the tool's effectiveness, conduct a "controlled experiment":
- Select a specific theme/question that needs to be answered as part of the strategic planning process (e.g., what will be the likely impact of emerging market growth to our business model? how will the competitive environment be changed as a result of a stronger role of states and regulation?)
- Set up two groups that are comparable e.g. in terms of their size, diversity, and skill sets. In one group, deploy the Deliberatorium to surface perspectives on the strategic planning question; in the other group, run the process in the traditional form.
- Compare the results in terms of (i) quality/depth of the discussions and their output; (II) level of engagement/satisfaction in the process by the users, and (iii) organizational effort needed to run and harvest the deliberation.
3. If the results for the experimental group are better than those of the control group, further build out the tool and supporting processes, and deploy more broadly at the next opportunity.