Hack

Hack: Making Large-Scale Deliberations Better, Online: The Deliberatorium

by Mark Klein - Principal Research Scientist at MIT Center for Collective Intelligence

September 3, 2011 at 7:32pm

30 Comments

14 Ratings:

Overall 4.71
Innovative 4.71
Detail 4.71

Contribution Summary

Summary

The Deliberatorium is a software tool designed to help organizations better harvest the knowledge and incorporate the perspectives of their members to identify solutions for complex problems, avoiding the dysfunctional behaviors (such as noise, disorganized content, and polarization) that other social media often produce when applied to challenging topics.

Moonshot(s)

Problem

Decision-making in organizations rarely fully harvests the collective intelligence of its members, even for very high-stakes strategic decisions where such contributions can make the difference between disaster and success. Social media technologies (e.g. email, web forums, chat rooms, blogs, wikis, and idea forums) have the potential to address this critical gap but, while they have enabled an explosion in how much communities can weigh in on topics they care about, they often create more heat than light when applied to complex, controversial problems:

Scattered content: The content created by existing social media is typically highly scattered, so it's hard to find all the different contributions on a topic of interest. This also fosters unsystematic coverage, since users are often unable to quickly identify which areas are well-covered, and which need more attention.

Low signal-to-noise ratio. Social media content is notorious for being highly redundant, with potentially important contributions often drowned-out by repeated postings made by a small fraction of the community.

Non-collaborativeness. Social media systems often elicit many relatively small standalone contributions, rather than a smaller number of more deeply-considered ideas, because collaborative refinement is not inherently supported.

Polarization: Users of social media systems often self-assemble into groups that share the same opinions, so they see only a subset of the issues, ideas, and arguments potentially relevant to a problem. This tends to lead people to take on more extreme, but not more broadly informed, versions of the opinions they already had.

Dysfunctional argumentation: Existing social media systems do not inherently encourage or enforce any standards concerning what constitutes valid argumentation, so postings are often bias- rather than evidence- or logic-based.

Disorganized content: Existing social media generally create very disorganized content, so it's time-consuming to find what has been said on any topic of interest. This fosters unsystematic coverage, since users are often unable to quickly identify which areas aren't yet well-covered and need more attention.
Low signal-to-noise ratio. Social media content is notorious for producing highly redundant content, so enormous effort is typically required to "harvest" this wisdom to inform better, more broadly-supported decisions.
Quantity rather than Depth. Social media systems often elicit many relatively small contributions, rather than a smaller number of more deeply-considered ideas, because collaborative refinement is not inherently supported.
Polarization: Users of social media systems often self-assemble into groups that share the same opinions, so they see only a subset of the issues, ideas, and arguments potentially relevant to a problem. People thus tend to take on more extreme, but not more broadly informed, versions of the opinions they already had.

Here's just two examples (see this blog post for more): (1) Intel ran a web forum on organizational health that elicited 1000 posts from 300 participants. A post-discussion analysis team invested over 160 person-hours to summarize these contributions (at 10 minutes a post, probably longer than it took to write many of them in the first place). The team found that there was lots of redundancy, little genuine debate, and few actionable ideas, so that in the end many of the ideas they reported came from the analysis team members themselves, rather than the forum. (2) Google used their "moderator" system to collect ideas for which charitable projects the company should fund (project10tothe100). The company had to recruit 3,000 employees to filter and consolidate the 150,000 ideas they received in a process that put them 9 months behind their original schedule. The vast majority of these ideas were minor variants of simple suggestions (e.g. support public transport, make government more transparent, and so on). Surely that vast amount of effort could have been used to compose a smaller number of more deeply-considered ideas, rather than many shallow ones.

This introduces a quandary. We want as broad participation as possible, since that increases the chance of finding a truly transformative idea, but current social media technologies often capture only a fraction of the collective wisdom of a community, when applied at scale, and harvesting this wisdom can be an expensive proposition.

Solution

The Deliberatorium is a software system, developed at the MIT Center for Collective Intelligence over the last five years, designed to transcend these limitations and realize the enormous potential social media has for enabling better organizational decision-making. The approach is simple. Members of a community are asked to make their contributions in the form of a deliberation map, a tree-structured network of posts each representing a single issue (question to be answered), idea (possible answer for a question), or argument (pro or con for an idea or other argument):

A screenshot from the Deliberatorium. Each line in the left pane represents a single issue, idea, or argument. Each such post can have it's contents viewed, edited, discussed, and rated in the right pane.

The community follows a simple process to ensure that the deliberation map is as useful as possible:

Every author starts by "unbundle-ing" their contributions into individual issues, ideas, and arguments, and adding their unique points to the relevant part of the map. A key tenet is the "live and let live" rule: if you disagree with an idea or argument, you shouldn't change the post to undermine it, but should rather create new posts that present the strongest ideas and (counter-)arguments you can muster, so all contributions can compete on an even basis.

Moderators help ensure that these guidelines are followed. New posts can, at first, only be viewed by moderators. When a moderator verifies that a post follows the deliberation map guidelines, they can be viewed, edited, commented, and rated by the full community. If a post doesn’t yet meet the guidelines, the moderator leaves comments explaining what needs to be done to fix them. Moderators play a modest “honest broker” role: their job is not to evaluate or change the content of a post, but simply to help authors ensure that the content is as accessible as possible to the community at large. Moderators can check each other's work, as they do in such systems as Wikipedia and Slashdot, to ensure they are doing a good job.This process is supported by such software capabilities as open editing (any user can check and improve posts), watchlists (which automatically notify users of changes to posts they have registered interest in) and version histories (to allow users to roll-back a post to a previous version if it has been "damaged" by an edit). The system also provides a powerful set of attention allocation metrics to assess how well each part of the deliberation is going, and thereby to help community members focus their efforts where they can do the most good.

Deliberation maps have many advantages over conventional social media. Every unique point appears just once, radically improving the signal-to-noise ratio. All posts appear under the posts they logically refer to, so all content on a given question is co-located, making it easy to find what has and has not been said on any topic, fostering more complete coverage, and counteracting polarization by putting all competing ideas and arguments right next to each other. Careful critical thinking is encouraged, because users are implicitly encouraged to express the evidence and logic favoring the ideas they prefer, and the community can rate each element of an argument individually. Users, finally, can collaboratively refine proposed solution ideas. One user can, for example, propose an idea, a second raise an issue concerning how some aspect of that idea can be implemented, and a third propose possible resolutions for that issue. The value of an argument map can extend far beyond the problem instance it was initially generated for: it represents an entire design space of possible solutions that can be readily harvested, refined and re-combined when similar problems arise at other times and places.

To help make this more concrete, here's a short video of a fictional, but illustrative, use scenario:

I picked a simple example for the video that I thought would be accessible to most people, but deliberation mapping can of course be applied to all sorts of decision problems, including major "management processes" such as strategic planning, budgeting/resource allocation, risk management, product development, and so on.

The current user interface, while functional, obviously has an "old world" feel. This reflects my limitations as a programmer, as well as a strategic calculation. I've noticed that many hugely successful web 2.0 systems (such as email, news groups, Wikipedia, and Facebook) became that way despite having initially quite rudimentary user interfaces. The key success factor often seems to be the compelling-ness of the idea, rather than the polish of the implementation. I've focused, therefore, on rapidly exploring new ideas, rather than polishing old ones. Going forward, my hope is that the most powerful ideas explored in this work will be taken up by commercial vendors and incorporated in their products.

Practical Impact

We've tried out the Deliberatorium in a range of business, government, educational, and other contexts. Our experience to date is that it can make a real difference. Our first experiment involved translating a web forum on carbon offsetting (hosted on planeta.com on May 2008) into a deliberation map. The original 13 page discussion, filled with the digressions and repetitions that typify web forums, translated into the following eight item deliberation map:

The carbon offsetting web forum summary map

This was a startling illustration of the potential of deliberation maps for harvesting a community's collective knowledge in a way that is qualitatively more useful than conventional social media.

Our first large-scale evaluation was at the University of Naples, where 220 masters students in the information engineering program were asked to weigh in, over a period of three weeks, on what use Italy should make of bio-fuels. All told, the students contributed nearly 2000 posts, creating a map that was judged by content experts to represent a remarkably comprehensive and well-organized review of the key issues and options around bio-fuel adoption, exploring everything from technology and policy issues to environmental, economic and socio-political impacts.

A small portion of the Naples bio-fuel deliberation map

We found that, initially, about 2/3rds of user posts were structured correctly as originally created, and this increased to about 85% by the end of the deliberation. Also, the remaining posts almost always just required some simple fix e.g. people would call it an "idea" when it should have been a "pro" (see this paper for details). Overall, we were encouraged by how well people could use the deliberation map structure. It made it possible to support the user community with the part-time support of just two moderators. We were hard-pressed to imagine any other approach that would allow over 200 authors to write what was in effect a substantial book on a complex subject, in a couple weeks, with no one in charge.

Our first business-centric evaluation was conducted with Intel Corporation on the question of how "open computing" (i.e. where users are given greater access to computing tools and data) should be used in the company. Contributions were purely voluntary. A single moderator was able to support the discussion with very little effort. The end result (see below) was that Intel received a substantive and well-organized overview of important issues in this space from 73 contributors, including many from outside the company, at close to zero cost.

Map generated for the Intel deliberation on open computing

We've also conducted evaluations with the US Bureau of Land Management and the University of Zurich, among others, and we've learned that, when compared to conventional social media, it can help you get better content on how to solve complex problems, at lower cost, small voices can be heard and, perhaps most importantly, it's much easier to find the "good stuff".

Challenges

The Deliberatorium is not always going to the best way to harness an organizations' collective intelligence - it really depends on the nature of the problem and the organization itself. We've identified the following attributes that seem to signal a "Deliberatorium-ready" challenge:

complexity: The problem inherently involves a large space of possible solutions and many evaluation considerations.
creative vs competitive: The problem is predominantly creative (which offers maximum room for new win-win solutions) rather than purely competitive (a zero-sum struggle around the allocation of limited resources).
distribution: There are many participants who are geographically and/or temporally distributed, so face-to-face and teleconferencing solutions are not practical or satisfactory.
barriers: There are no systemic (e.g. intellectual property, organizational, or cultural) barriers to collaborative decision-making.
Critical mass: There is a critical mass of users with expertise on the problem and a desire to help solve it, even though they may disagree strongly about what solutions to adopt, or even about what constitutes a good solution.

Many organizational problems satisfy the first three criteria, so we will focus our attention on the latter two.

Systemic barriers: One of the largest barriers to adoption of any web 2.0 technology is that the current organizational structures and incentives are not collaboration-centric. Managers, for example, may be unwilling to limit their leeway by adopting a more open decision-making process. In this atmosphere, other members of the organization are unlikely to put their heart into exploring and critiquing new solution ideas. This is a cultural rather than a technological problem, and the solution involves some combination of grass-roots push and visionary leadership.

Critical mass: Any web 2.0 system requires a critical mass of committed participants to be successful. If people sense that others are not participating, they won't either. The most common question people ask about the Deliberatorium approach is: given that it requires more up-front effort from participants than other social media, will it be able to motivate and support really large-scale deliberations? Let's look into that.

Active participation in social media systems tends to occur when the benefits, to the individual contributors, substantially exceed the costs. With small numbers of participants, clearly an informal approach, based for example on phone or email or web forums, minimizes participation costs and is capable of producing good outcomes. But this picture, we suggest, changes as the scale of the discussion grows.

Users contribute to social media systems predominantly for two reasons: (1) t) to become a hero (have a substantive positive impact on a community they care about), and (2) to find their tribe (i.e. get connected with people who share their interests). How does this play out in a deliberation context? Let us make the reasonable assumption that points (individual issues ideas, and arguments) have an uneven distribution in the user population: some points are known to most people, some to only a few. We can thus expect widely known points to be submitted frequently from multiple sources, and more “out-of-the-box” (but potentially valuable points) to arise less often. It seems clear that the number of unique points contributed to the deliberation will grow much more slowly than the number of participants. Our simulations suggest that this growth is roughly a logarithmic function of the community size, implying a roughly linear growth in redundancy as the community scales.

Simulation results for number of redundant posts in a deliberation.

The larger the user community, therefore, the more potential redundancy there is, and thus the more value argument mapping offers by eliminating that redundancy. There is widespread dissatisfaction with the low signal-to-noise ratio of current social media. We can expect that, as the scale of the discussion grows, users will increasingly recognize the opportunity to “become a hero” by contributing something (i.e. creating a high-value solutions map) that is highly valued by the community. Argument mapping also increases user’s chances of “finding their tribe”. While contributing to unstructured discussions is easier, the high volume and redundancy of such discussions means that most posts will probably be overlooked by most readers. In an argument map, by contrast, if you have a unique point to make, it has a much greater chance of being seen. We can thus expect the benefits of argument mapping, to a contributor, will increase rapidly with the size of the user community.

We can also expect, moreover, that the costs of participation for a contributor will grow only slowly as the community scales. Contributing to a deliberation map incurs two main costs:

unbundling the contribution into its constituent issues, ideas, and arguments
locating the proper place for these elements in the map

The cost of unbundling a contribution is independent, of course, of the size of the map. The cost of locating a contribution should increase with the size of the map, but only slowly. Remember that a deliberation map is structured like a tree. To find the right place to put a post in a tree, you just have to pick the right top-level branch to place it, the right sub-branch under that, and so on, until you reach the place where it belongs. If the average branching factor (number of sub-branches per branch) of a tree is N, then the average number of steps needed to locate a post is just the Nth logarithm of the tree size. The overall picture is thus the following:

Qualitative model of cost vs benefit tradeoffs for contributing to an argument map as a function of the size of the active user community.

The benefits of adding to an argument map grow rapidly as the community scales. The costs of adding to an argument map grow only slowly with scale. At some point, we can expect, the benefits to authors will greatly exceed the costs, thereby providing compelling incentives for participation.

What about moderation? Who will do it? Will we be able to find enough moderators? Power users, with a track record of successful argument map creation, can be recruited to join the moderator pool. We estimate that there needs to be about 1 moderator for every 20 active authors, to ensure that posts are checked and certified in a timely way without burdening the moderators too much. This figure is well within the bounds of the number of “power users” that emerge naturally in web 2.0 systems. Scale helps us here too. The number of potential moderators should grow with the size of the community, but the number of posts to certify should increase a lot slower than that, as people increasingly find that their points are already captured in the map. So the per-moderator burden should decrease as the community scales. A final key point is that an organization will probably prefer to make a relatively modest investment in up-front moderation in order to avoid the much larger costs of harvesting a discussion after it has occurred.

First Steps

Interested in trying out the Deliberatorium to support decision-making in your organization? The system is publically accessible at http://deliberatorium.mit.edu/ You can look at some existing maps, play around in a "sandbox" to see what authoring is like, and even start up your own deliberation map if you like (contact me if you want to do the latter). If you want to run the system on your own servers, the source code is in the public domain (again, contact me for that).

Our experience from previous evaluations suggests that something like the following process is a good way to go:

1. Embed the tool into a management process so that it's part of the "workflow" and taken seriously (for instance, make it a part of the strategic planning process). Your organization's members are much more likely to try a new approach if it is part of a conversation that matters.

2. As an initial test of the tool's effectiveness, conduct a "controlled experiment":

Select a specific theme/question that needs to be answered as part of the strategic planning process (e.g., what will be the likely impact of emerging market growth to our business model? how will the competitive environment be changed as a result of a stronger role of states and regulation?)
Set up two groups that are comparable e.g. in terms of their size, diversity, and skill sets. In one group, deploy the Deliberatorium to surface perspectives on the strategic planning question; in the other group, run the process in the traditional form.
Compare the results in terms of (i) quality/depth of the discussions and their output; (II) level of engagement/satisfaction in the process by the users, and (iii) organizational effort needed to run and harvest the deliberation.

3. If the results for the experimental group are better than those of the control group, further build out the tool and supporting processes, and deploy more broadly at the next opportunity.

Helpful Materials

Want to learn more about the Deliberatorium? You can visit our web site for downloadable papers, a short introductory video, or a link to the system itself. You might also want to visit the home page for the MIT Center for Collective Intelligence, where the Deliberatorium project makes its home. Finally, feel free to contact Mark Klein, the project lead, if you have any further questions.

Documents:

guidelines.pdf

Images:

No images at this time

Videos:

Input

Comments

Sean Schofield

February 28, 2012 at 12:37pm

Hi Mark,

A few have the raised the point of "user search cost" being a barrier to usage. Presumably, this cost is positively correlated with the quantity of content. More stuff = more stuff to process in order to find where each piece fits best.

If the presumption is true (and we assume people care about respecting the best fit rule - probably safe to assume if there is a visible norm), then this is incrementally more threatening as the number of "deliberators" increases - which as I understand it, is what you ideally want to do.

Assuming this hasn't been solved...

Random thoughts:

1) Could some kind of text analysis tool be used suggest where to place an author's submission? (I.e., after writing a contributor the author hits a kind of "place me" button and it offers probable matches)

2) Could there be a kind of "fast pick" system, where as the user types their contribution a live filter narrows down possible locations based on key words or phrasing.

3) Could a choice driven system narrow down the "box" the contribution is likely to do in. I.e., the deliberatorium provides a decision tree and poses yes/no questions to the author, allowing the author to quickly filter out the non-relevant branches.

4) More of a stretch...if there is a natural "law" about the nature of progression (i.e., pattern of complexity) of a deliberatorium, then you could try to predict where the post is most likely to belong to in combination with any of the above.

Thanks for reading,

Sean

Mark Klein

March 5, 2012 at 4:03pm

Sean,

Thanks for your comment, some great ideas that I intend to explore more. Here are some more detailed responses.

Yes, that is a key issue, though the search cost should, in theory, only grow as a logarithmic function of the size of the argument map, which helps.

I think that makes a lot of sense. I've tried that out a couple years back, but I wasn't happy with the retrieval accuracy and completeness at that point, so I disabled it until I could implement a better approach. Currently, I'm thinking of using a variant of latent semantic indexing that takes advantage of the argument maps' tree structure to help improve the hit rate.

2) Could there be a kind of "fast pick" system, where as the user types their contribution a live filter narrows down possible locations based on key words or phrasing.

Interesting idea, like the google live search thing.

That's also interesting! But let's say the system, instead of following a binary decision tree, provides a list of the words whose selection would, at each point, most rapidly narrow down the range of possible matches, based on what you've already entered. Then you might not even have to type a lot of the time, you could just select words from the list suggested by the system.

I think that would help as well. In an argument map, people will often add posts in a top-down manner, attaching new posts under the posts they've just recently created or edited or rated or viewed. We can use that insight to refine the list of suggestions that the system makes.

Again, thanks for the suggestions, appreciated.

Mark

Mark Klein

September 14, 2011 at 10:19am

Michael Harris (Michael.J.Harris@Intel.com) made an interesting point I wanted to share with the MIX community:

"Some research suggests independent brainstorming followed by collaborative sharing produces much greater diversity and quality of ideas than the method we typically use at Intel which is immediate, collaborative brainstorming.

The limitations of typical team brainstorming include free riding, evaluation apprehension, productivity blocking and conformity effects. I would think these same limitations could apply to online settings in which all participants have immediate access to all content.

Given the primary objective is innovation (with redundancy reduction as a necessary but secondary objective), what strategies might be applied to the deliberatorium to combat these types of limitations? For example, could you require participants to make a contribution before accessing others’ content and organizing their submission?"

Mark Klein

September 14, 2011 at 10:20am

Here was my response:

That's a great question. My understanding is that the problems you describe apply mainly to *face-to-face* brainstorming sessions. In such meetings, only one person can talk at a time, and people typically self-censor to deal with that bottleneck, e.g. based on their perception of the power dynamics in the room. I think that these problems are *much* less of a factor in asynchronous online brainstorming, since everyone can make contributions in parallel without taking away airtime from anyone else. That really seems to take the lid off. In fact, the problem often becomes "too much" participation, rather than too little. The judicious use of anonymity can also help with self-censoring. The Deliberatorium, for example, displays the ratings for posts but not *who* made those ratings, making it more comfortable for people to tag less-promising posts and thereby separate the wheat from the chaff. Some researchers have also made a case for allowing users to use pseudonyms for their account names, there are arguments back and forth on that.

In any case, I think it's a great idea to ask that people brainstorm independently before entering stuff into the deliberation map, it can probably only help. I'm going to make a note of that idea for future reference, thanks.

Mark

Jerome Gayerie

September 5, 2011 at 1:34pm

I was pleased to collaborate with Mark Klein last year, submitting some ideas to him to improve the deliberatorium. And for managing our exchanges, we used.... the deliberatorium itself. So the deliberatorium is not only useful for large groups, as in Naples or Intel but also for very small groups (two persons here).
This is a great tool to create content, store, organize and make decisions. It is based on the simple and powerful principle of issue-maps, enriched with many useful features (comments, search, tracking, rating, version management, etc ...), all in an effective user interface.
Of course the tool is not everything and the methodology (guidelines) is as important as the tool itself. Again, guidelines are common sense and efficient. All this allows to get easily a result content that is both detailed and clean. Particularly because the content is separated from the comments that improve the content (unlike a forum where content and comments are mixed).

By the way, guidelines could also be a subject of deliberation, open to users. This would enrich the guidelines with good practices and different ways to use the tool.

I do not know if it was experimented before, but the deliberatorium may be useful for supporting real meetings (in the same room or conference call): people discussing while the person in charge of organizing the debate, minutes the content in the deliberatorium. Of course, here, the frequency is limited by the fact that only one person speaks (and one person write) at the same time. But on the other hand, the clarification of ambiguous points would be faster than during an online asynchronous exchange. For larger groups, one could even imagine sub-groups working concurrently in a first step. A second step would then be necessary to merge the whole.

Mark Klein

September 5, 2011 at 1:43pm

Jerome,

Thanks for the comments! Deliberation mapping has in fact been used to support synchronous meetings. It can work quite well if you have someone who is mentally agile enough to translate a discussion into a map in real time. I think another good option is the idea of interleaving asynchronous building of the deliberation map with synchronous discussions about the map. You can get breadth of input from the parallel asynchronous contribution phase, and also benefit from the consensus and shared understanding development that can emerge from synchronous discussions. I've used that approach frequently myself, most recently in a project with Siemens.

Mark

Hajo Reijers

September 5, 2011 at 12:21pm

In my view, the deliberatorium packages powerful concepts in an attractive and practical form for collaborative problem solving. While the current interest in sharing content and engaging in discussions over the Internet is tremendous, we need yet to experience the true impact that cooperation between independent and knowledgeable individuals can make. Through its emphasis on condensing the discourse and, in this way, making it accessible and meaningful to all discussants, I think the deliberatorium has the potential to create that.

Hajo

Cristiano Ferri Faria

September 5, 2011 at 7:08am

Deliberatorium is a really interesting project. I have been following it last months as well as Mark Klein’s research progresses. I also work in a large-scale deliberation project, called e-Democracia (www.edemocracia.gov.br). Created in the Brazilian House of Representatives, it’s an official website whereby people can share their opinions, critics and suggestions to legal texts with representatives.
The major challenges we have here in e-Democracia is these same very well-defined (by Mark) problems in large-scale platforms for debate: Disorganized content, low signal-to-noise ratio and quantity rather than depth.
And the solution, with Deliberatorium, in which participants are invited to set their opinion, argument or idea in an argumentation map, is great and functional! In my view, there is a just challenge in this kind of system and I know Mark’s aware of it.
It demands certain initial effort from the participant to read and search in the argumentation map where his point could be inserted. This effort may discourage participants who want to contribute but haven’t great technical knowledge about the subject, so it will be slightly costly to search where his contribution should be included. Moreover, there is a kind of participants who just intend to present a testimony but an argumentation map may seem not appropriate or open to this kind of contribution. One possible solution for that is just let people to participate the way they prefer, so the moderator will have to work harder to organize it.
But I’m sure Deliberatorium is dealing well with these challenges. In my opinion, this is one of the best solutions in this field by far.
Cheers Mark

Cristiano Ferri Faria
Brazilian House of Representatives’ e-Democracia Project
@cristianofaria
@edemocracia

David Price

September 4, 2011 at 1:25pm

Among the excellent set of finalists, the two with which I'm most familiar – and admire greatly – are the Deliberative Corporation and MIT's Deliberatorium.

Both are grounded in a systematic form of deep and wide listening, informed structured dialogue, mutual understanding, and pragmatic negotiation geared towards effective action: and both point towards a more open, agile, rigorous and transparent system of civic and corporate governance.

Both seek to build on social media’s strength as a serendipity catalyst by giving participants the means to interact mindfully, to surface difficult questions, to learn collaboratively, creatively and cumulatively from each other, and to take responsibility and hold each other accountable for the decisions arising from their deliberations.

The deliberative work that both entail is demanding and inspiring – as the best work should be – and in an age of eddying noise and complexity, of financial and political uncertainty, and of intense pressure on resources, this work has never been more important.

Both would be worthy winners of the Management 2.0 Challenge.

Mark Klein

September 3, 2011 at 10:32am

I got the video done and have added it to the post - hope you find it helpful.

Mark

ana bazzan

September 2, 2011 at 1:37pm

I heard about the Deliberatorium during a talk given by Mark Klein as an invited speaker in an AI conference. I immediately though that a) we could use it at our department to reach some consensus regarding more or less contentious issues, and b) this could be a nice application to combine with other AI techniques in order to try to ease the role of the mediator.

So far we are selecting a topic to discuss using the Deliberatorium and I have also recommended the site / the work to a PhD candidate. He is thinking about combining this tool with recommender systems in order to guide new users. I am confident that this will be a nice work and I hope we can work together with Mark in this direction.

Ana Bazzan

Robert Laubacher

September 2, 2011 at 12:37pm

For MIT's Climate CoLab project http://climatecolab.org/, I've had occasion to use output from the Deliberatorium.

As part of Mark's research, he and a colleague oversaw the development in the Deliberatorium of an argument map that outlined the range of possible approaches for addressing climate change.

Mark sent me a link to this argument map, and I found it to be quite impressive. It nicely captured the broad range of ideas that I had seen expressed, albeit in scattered and disorganized form, in the many books, articles, and web sites I'd been reading on climate change.

This map has become a valuable resource to our team as we think about new ways to organize activities on the Climate CoLab in coming months.

And we have implemented a somewhat simplified version of the Deliberatorium in the Climate CoLab, which allows members of our community to comment on and add to mini-argument maps that address specific issues in the climate change debate.

For an example, see http://climatecolab.org/web/guest/plans/-/plans/contestId/4#plans=subview:issues

Adriana S. Vivacqua

September 1, 2011 at 5:17pm

The Deliberatorium is a real step towards collaborative deliberation. I particularly like how the IBIS structure fits into the picture, providing an organization for ideas.

Breaking ideas into smaller elements is key to making individuals think about their posts and structure them correctly. I wonder if users will have difficulty breaking their ideas into the three elements proposed, and what to do if these elements are linked together? Do you know if users have problems with the structure and rules at first? It should take them some time to properly learn what and how to send their ideas and opinions.

I also worry that, with many posts, the tree might become too large, and cumbersome to read through, making it hard to find the appropriate location for a post. Even though you claim it won't, when I think of a service such as twitter, I can't help but imagine what an argument tree would look like (especially if some tweets had to be broken into 2 or 3!) This may become a problem for users. Perhaps some sort of search functionality would help?

All in all, a well rounded and thought out initiative, which will probably benefit many organizations!

Mark Klein

September 3, 2011 at 10:21am

Adriana,

Thanks for your thoughts. In our Naples evaluation (our largest to date), we found that, initially, about 2/3rds of user posts were structured correctly as originally created, and this increased to about 85% by the end of the deliberation. Also, the remaining posts almost always just required some simple fix e.g. people would call it an "idea" when it should have been a "pro" (see this paper for details). So, you're right, not everyone got it perfect at first, many people initially treated the deliberation map as if it was a web forum, but overall we were encouraged by how well people could use the deliberation map structure.

You're also right that complex topics can lead to big maps. The Deliberatorium does include a search capability to help people quickly find where to place their contributions.

Mark

Ali GURKAN

September 1, 2011 at 12:01pm

What I liked most about Deliberatorium the possibility to use it a wide range of areas with minor modification. Although its use should not limited to management purposes, it has one (but a "big" one) advantage over other advantages: it's cheap. There is no regular business trips necessary to gather people together. Well, one may also add that according to his own will. Even so, that shows the flexibility of the platform.
At another presentation I remember seeing the idea of using a random populations for rating. That would work here as well for preventing "friendly" ratings.
Another thing might be limiting posts to a certain number of characters like on twitter. Eliminating endless user comments right from the beginning would also decrease redundancy as people - I assume - are more likely to read brief comments. That's also somehow the idea behind argumentation: Disintegrate the parts of a whole as an idea, a pro or a con argument.

Mark Klein

September 3, 2011 at 10:30am

Ali,

Thanks for your note. I agree, having concise posts for each issue, idea, and argument should help make it easier to quickly get a sense of the key points in a deliberation map. Currently, we achieve this by having length limits for the titles of posts, allowing people to make the post descriptions as long as they need. But maybe it will make sense to also have length limits in the descriptions, something to think about.

The idea of randomly assigning people to rate posts to avoid "gaming" is also an interesting one, thanks.

Mark

Mark Klein

August 31, 2011 at 3:06am

I'd like to thank the MIX judges and community for many helpful comments on how to improve the hack. I made the following changes in response:

added examples and links to flesh out the problems current web 2.0 systems have with supporting complex deliberations
added a short video with a fictional but illustrative use scenario, so people can see the tool in play
discussed how to deal with potential challenges to using this approach, especially the critical challenge of working at scale

Mark

Daniel Bassill

August 29, 2011 at 7:28am

It seems that this could add value to community organizing and on-going problem solving if the facilitation team were sustained for many years. Have you done any experiments with using this as a project management tool? For instance, if you put a set of blueprints for building a hotel on line, you could set up a template for discussions starting with the initial design and financing work through the first steps of laying the foundation through the final step of opening the doors to new customers.

If another group of people wanted to build a similar hotel in a different places they could follow that template and even learn from the work done in earlier projects. I have used concept maps to describe strategies that would result in more and better non-school tutor/mentor programs in inner city neighborhoods. http://tinyurl.com/TMC-4-Part-Strategy

If discussion nodes could be attached to each of the steps of these maps, the facilitation could be a form of project management intended to get more people involved, informed and active in achieving the goals of the project.

Instead of starting a discussion with a blank sheet of paper we’re starting with a goal in mind and working backwards to fill in what we and others already know. As we do that we’re working forward to know more and act on what we know to make volunteer-based tutor/mentor programs available in more places, based on the strategy that is outlined.

Is this possible? Is it already being done this way?

Mark Klein

September 3, 2011 at 10:45am

Daniel,

Thanks for your comments, sorry to take so long to reply. I do think that a deliberation map for one project could add a lot of value to people doing later projects with similar goals. They could quickly see what options people had explored in the past, as well as what the pros and cons were. I haven't personally run this kind of multi-year scenario, but Jeff Conklin at Cognexus Institute observed (see this paper) that a major electric utility, over a period of years, frequently consulted maps from previous deliberations when facing similar challenges.

Mark

Catherine Spence

August 29, 2011 at 7:27am

Intel IT has been pleased to work with Mark Klein from the MIT Center for Collective Intelligence on a series of investigations aimed at exploring the concept of collaborative argumentation.

Our projects validated the potential of the combination of social media and argumentation. Social media enables the hosting of large scale discussions with a vast number of diverse users over the internet. Argumentation captures the logical structure of debates and encourages evidence-based, critical thinking with systematic coverage of topics.

One of the biggest benefits of Deliberatorium was the ease of generating the argument map. The “moderate-as-you-go” approach promoted structured problem solving during the ideation and saved time during the post-processing of content. The compact format was useful to reduce complexity and highlight key content and contributors. As compared with threaded conversations organized by time in other tools such as web jams and wikis, the argument map saved time and effort synthesizing the results. Further, the argument map provided a useful artifact for later data mining and as a historical record. Since we often go back and periodically reevaluate our position on a variety of topics, the argument map can be used as a starting point without recreating the same content on complex issues.

In terms of real world corporate application, we already experienced some technology transfer by influencing Intel’s internal ideation tool to include basic argumentation capabilities. These enhancements were incorporated in a successful pilot, which then led to a production deployment. Outside of Intel, we have debriefed our commercial social computing provider on the collaborative argumentation research and results in order to inspire future products which could eventually be consumed at Intel.

Congratulations to Mark on his excellent work!

Kartik Subbarao

July 14, 2011 at 1:57pm

I think that the Deliberatorium provides a valuable toolset that helps in managing large-scale decision-making discussions. I'd suggest the following to further increase the reach and capability of this approach:

+ Modularize and open-source the code so that it can be deployed alongside existing Wiki/collaborative systems. That way the user interface can be further refined and take advantage of other features. For example, if a social networking engine already has captured tags/reputations on people, these can inform the node ratings.
+ Partner with others in this space -- for example, I think it'd be intriguing to see Quora.com run some of their questions as Deliberatoriums, given the high quality of participants that they have attracted in various topic areas

While running a Deliberatorium on a given subject, I also think it might be useful to run a traditional threaded discussion in parallel, so that people can easily meta-comment in a freeform manner on the process itself, as the discussion is unfolding (i.e. I think that there are too many subtopics under this heading, we should break it up further, etc). This could be as simple as the "Discussion" tab in a Wikipedia page.

I think Deliberatoriums could be launched on a lot of consumer-friendly topics as well -- what type of car to buy, how to allocate a stock portfolio, etc. Here's where partnering with an existing knowledge-based service like Quora.com, Wikipedia, Yahoo Answers, etc would be particularly helpful.

Good luck to you guys on the MIX M-Prize!

Mark Klein

July 14, 2011 at 9:39pm

Kartik,

Thanks for the great suggestions, I'll give them some thought.

Mark

John Doe

July 7, 2011 at 10:06am

I would be curious of the high level view of this component in a holistic collective knowledge platform.

This seems more like an experiment fit for and limited to academic practice (or an innovative R&D group at a corporation) when in isolation. Would I be correct to assume this is indeed a component and theory being developed as one part of a larger effort to develop collective intelligence platforms at MIT?

It seems to me that manual argument mapping through a proprietary system is not the best approach, considering the user education scenario. It is very easy to begin this exercise and maintain a high level of quality for small groups of highly educated and motivated people. But given an implementation of a wider audience, the demand on the moderators go up, the quality of content goes down and the receptiveness to yet another process usually hinders if not thwarts the effort.

Efforts like Vulcan's Evri, Halo, Microsoft's PowerSet and other such companies have proven that you can get the logic of an argument through NLP methods. This allows you to build truth tables and argument maps of large bodies of text. These methods are so accurate, they can "learn" from the text books and pass an AP exam. Another more well publicized version of the same technology is IBM's Watson.

While utilizing the social platforms with only what they offer, I believe you are indeed correct in saying that they have the downfalls listed in challenges. But to approach the issue by building another platform is a bit of an uphill battle. If we truly would like to unlock the potential of collective knowledge, I think the NLP route is a much more viable solution. This solution takes care of all of the downfalls and can even link out of dialogue logic and reason to the current issue at hand. It also allows the group to take non-user generated content from the ages and take that information into consideration.

Can you let me know if you've found this not to be the case and what benefits this hack provides over the NLP/collective knowledge linguistics method?

It is an interesting piece of work and fits well into the larger picture of collective knowledge linguistics and understanding. MIT seems like it is always nourishing the latest and greatest ideas. Bravo!

Mark Klein

July 7, 2011 at 8:38pm

Anthony,

Thanks for your comment.

Yes, I do see the Deliberatorium as part of a larger effort to enable large-scale collective intelligence. The Deliberatorium is aimed at helping communities rapidly and effectively define a space of possible solutions for a complex problem. I have a separate project (see http://cci.mit.edu/klein/negotiation.html) aimed at helping decision-makers rapidly search such spaces for pareto-optimal (win-win) solutions.

My goal, however, is to make this approach useful for large diverse user communities, not just for small groups of researchers or the like. In the evaluations I've done to date, the participants were a pretty mixed bunch, with as many as 200 working together. My hypothesis is that the cost-benefit ratio for the Deliberatorium approach will actually improve with the number of participants. The number of potential moderators, for example, should scale linearly with the size of the community (typically a few percent of the authors in a social media community are interested in being "super-users"), but the number of posts to certify should increase a lot slower than that, as people increasingly find that their points are already captured in the map and they don't need to create new posts. So the per-moderator burden should decrease as the community scales. My goal is to test this hypothesis by running deliberations of increasing scale.

My understanding is that NLP technology is not yet capable of understanding text well enough to summarize the key issues, ideas, and arguments on a complex topic. Watson, for example, doesn't summarize anything, it relies on finding an existing snippet of internet content that answers a simple focused question. When creating a deliberation map, most of the questions are unknown at the start and are generated as part of the deliberation process. Evri and Powerset are also I believe used for search, rather than summarization. Could you point me to documented examples of software generating something like deliberation maps?

Even if/when NLP technology overcomes this hurdle, deliberation maps are still potentially useful because they help make deliberations more complete. In a deliberation map, it's easy to see where there are gaps (e.g. when an idea has few arguments attached to it) and we can direct the communities' attention to filling those gaps in.

Incidentally, I am now looking into whether we can use crowdsourcing (e.g. based on Mechanical Turk) to moderate or even build deliberation maps. At least in the short term, this seems much more viable to me than an AI-based solution.

Regards,

Mark

John Doe

July 8, 2011 at 10:18am

Nice, thank you for the links. You are working on some very interesting topics.

It will be interesting to see how it plays out and the usage once a large scale pilot is done. I hope we will have visibility into the usage statistics. I will be quite interested to see who should break up with who and why and how it is moderate once this hits Hollywood.

As I mentioned, Project Halo is a good example and has some good links (check out SILK). Also, I believe Evri demo'ed this type of functionality at an MIT event here in Seattle. I believe the query was "Who did Google acquire?" and it brought up fairly accurate results.

You are correct. The industry is misusing the IP and applying new technology to solve an old problem. I guess they didn't catch Clayton Christensen's work? Vulcan has been doing some great work in this area, although I'm not sure they found the proper way to bring this to commercialization on a conglomerate level scale. Their consumer products have been entertaining though.

Crowd sourcing is a great approach. Bravo on your work. I'm glad to have been able to discuss a little bit of what's going on with the top industry's top innovators.

John Doe

July 8, 2011 at 10:48am

Here is a quick overview of the process that we used to build a limited system at a past venture I was involved with:

Get the Stanford NLP processor.

Create an application that pulls 10 Ks from SEC or patent information off of USPTO.

Create a process that uses the Stanford NLP processor to tag the content.

Break up the content based upon different proximities (0 for word, 1 for section of sentence, 2 for sentence, 3 for paragraph, 4 for page, 5 for work of literature, 6 for corpus, et cetera).

Begin inserting the works in the following fashion (through a parsing process):

Word

WordID

Word

Part of Speech

POSID

POS

Word_POS

ContextID

POSID

WordID

Section

SectionID

Footprint (Contains delimited strings of the ContextIDs)

NextSectionID

NextSectionProximity

Once you have this, you can begin to build query footprints. For example, if you want to see who Microsoft's potential competitors may be, you can query for something like the following:

Microsoft [Proper Noun type POS] competes [verb type POS, proximity of 2 for query type] _______ [A loose pivot on a proper noun with a proximity of 2 from competes]

This is the first abstraction of the system, which unlocks a bit of I believe what you are saying. Instead of a statically generate deliberation map, you have a dynamic result based off of your large data set. It is of course over simplified, but the next level of abstraction would be doing synonym analysis for another pivot type (include all results for synonyms as well). Then you have footprint identification, entity extraction and other technologies that compliment this approach quite well. I believe this is how Powerset enabled the "In what year was Albert Einstein born?" type of scenario where it would give you your precise answer. From what I understand, this would be very similar to sending a query of, "What are the possible economic models that could be applied to providing a perceived incentive for reducing carbon emissions?"

I know there are some issues with it right now, but it seems like that is where the future is headed. It was interesting to see this approach to unlocking large data, versus the NLP type approach. I'm no expert, but perhaps some food for thought?

David Sims

July 1, 2011 at 11:26am

Hi Mark,

Great hack, thanks for submitting it. This is an critical area for those of us (almost everyone, I guess?) who work in teams, distributed around the globe, and figuring out how to move forward through online channels. I had a couple of questions on aspects of this that seem critical in distinguishing it from a listserv or Wiki, things you've touched on here but I'm curious to learn a little more.

I'm curious about the process behind Argument Mapping, and related, the role of the moderator. Seems like a lot depends on how the argument gets framed. Are there guidelines for how a whole raft of comments (like the one you took on with planeta.com) get categorized? How much does that rely on the moderator's judgment, or do you use anything like a tag cloud to group content under headings?

Also, as an editor, I'm wondering how you "export" the learnings or main ideas from this kind of environment in practice. In the implementations you've tried, does the moderator create a report or powerpoint presentation highlighting the arguments? Or is there something more automatic that can come out of this.

Again, neat idea. Would be interesting to see a screencast of it in play.

-- Dave

Mark Klein

July 3, 2011 at 2:06pm

Dave,

Thanks for your comment.

Framing: The framing of a deliberation (i.e. the issues in the deliberation map) can of course have a substantial impact on where the deliberation goes. We generally recommend that the customers for a deliberation start by defining, when they can, a top-level issue "skeleton" following guidelines detailed in the system help page. Open-ended issues (e.g. "how can we resolve climate change?") are, for example, usually better than yes-no questions (e.g. "should we use technique X to resolve climate change?").

Moderation: The system includes a search function to help authors find the right place to put their posts and, in our experience to date, they typically do a good job of that, without the need for moderator input. We did find, in our largest evaluation, that some of the map branches became pretty "bushy" (i.e. some posts had a lot of children under them); for those situations, we recommend that the customer, with a moderator's help, add new issues and ideas to cluster the children posts, so each post in the map has no more than about 7 children under it (the rough size of human short-term memory).

Exporting: the system allows users to write "guided tours", i.e. narratives made up of text interspersed with links to posts in the map. Each guided tour is thus a different summarization of the results of a deliberation. One could develop algorithms to *automatically* generate such tours (e.g. by doing a breadth-first traversal of the top-rated ideas in a map), though I think that people will generally create more compelling narratives than simple algorithms like that.

I'll look into creating a screencast, thanks for the suggestion.

Mark

David Rader

July 8, 2011 at 10:57am

Guided tours through a deliberation is an excellent idea. Would be very helpful as a teaching aid -- especially for social sciences. Can see applications to literature; maybe even music and visual arts.

Visual annotations would be helpful such as colors or line thicknesses to show which branches have recent activity.

Would be interesting to run controlled experiments on the quality of discussions given different structures. For example, on the deliberation on responses to climate change, one structure might focus on actors (national governments, local/city governments, large (mutliple-geography) businesses, small (local geography) businesses and individuals) versus a product structure (industrial goods, consumer goods, food, services, etc.). A panel could evaluate the breadth, specificity, usefulness, etc of the ideas generated in the two structures.

Are there publically-accessible implementations?

Mark Klein

September 3, 2011 at 10:59am

David,

Thanks for your comments, sorry to take so long to reply, I spend the last two months traveling on three different continents.

The Deliberatorium does include a "highlighting" capability that visually highlights posts in the map according to criteria you select e.g. it can highlight the branches which have been most active, which are the most controversial, the most highly rated, and so on. You can also use the search function for the same purpose, e.g. you can find all the posts that have been active in the last few days. I'm sure there's a lot more that can be done with that, though.

We did ran an evaluation comparing the quality of deliberations conducted, with the same topic and matched user populations, for a web forum vs the Deliberatorium. That data is still being analyzed: it's a big task, complicated by the fact the evaluation took place (at the University of Zurich) in German!

The Deliberatorium is publically accessible at http://deliberatorium.mit.edu/, and the source code is in the public domain. You can look at some existing maps, play around in a "sandbox" to see what authoring is like, and even start up your own deliberation map if you like (contact me if you want to do the latter).

Mark