Nachspiel at Polemarchus' » Methods in political science

Deliberating or quarrelling? Final draft of my thesis.

Sverre — Sun, 07 Nov 2010 14:00:33 +0000

After a long and arduous process, the work on my master’s thesis is finally nearing the end. Here is a slightly adapted version of the introduction, and a link to the print ready version (PDF).

Some of the inspiration for my thesis comes from an article in the student newspaper in Trondheim, Under Dusken, and similar comments over the following years. Political science professor Anders Todal Jenssen insisted that the student democracy in Trondheim lacked legitimacy because of the low voter turnout and that the introduction of political parties would be the solution to this problem. Binding platforms would make student politicians accountable to the voters and increase support for democracy. As a student representative myself at the time, I was provoked. We were proud of the lack of polarization within the student democracy and, although I didn’t know the term at the time, the level of deliberation. This started me on the quest for an alternative to Professor Todal Jenssen’s strong belief in the salience of political parties.

Democracy does of course seem unthinkable without political parties. Almost every democracy is dominated by a system of organized factions that structure, educate and drive the political process forwards. The necessity for such a system is no longer seriously questioned in political science. I do not believe, however, that any institution should be beyond question. Even if we have no intention to get rid of political parties, we should strive to understand the effect they have on democracy. As I will show in this thesis, one such effect may be reducing open and free deliberation among decision-makers. This may be a cost we are willing to pay, but not a cost we should pay without knowing its size. Deliberation should not be considered merely as a normative ideal for democracy, but also a descriptive model for understanding the workings of democracy. The amount and quality of deliberation may explain political decisions and outcomes that aggregative models do not. This should make deliberation a topic of interest even if one does not accept its normative justification.

Institutions influence the way democracy works. If deliberation is an important characteristic of democracy, we should take interest in how institutional design affects deliberation. There has been some research on this, but political parties, integral to almost all modern democracies, seem to have been neglected in this respect. I will show that there are sufficient theoretical reservations about their effect on deliberation that this should be a topic of proper empirical testing. To test the relationship between political parties and deliberation empirically, we need an approach for measuring deliberation. We should have a method with a theoretically sound basis, that measures what we want it to, and that is acceptable within the wider sphere of political science (a discipline that is both theoretically and empirically oriented).

I will examine various proposals for examining the amount and quality of deliberation and consider their respective strengths and weaknesses. The Discourse Quality Index seems to be the most promising such method in use today. I have tested the utility of the method for addressing whether political parties weaken deliberation in a political system. To do this I applied the method to two democratic bodies: the student parliaments of the universities in Trondheim and Oslo, Norway. Due to a limited amount of data I did not get significant results concerning the question itself, but I have collected practical experiences and new insight into the method and its applicability.

Quickly summarized, I find in my thesis that there seems to be sufficient theoretical grounds to support the assumption that political party systems are detrimental to deliberation. A major obstacle to empirical testing of this and other theories about deliberation is found in the current state of empirical methods. Several methods have been tried, but none seem to be able to completely combine the demands needed for the conclusions to gain general acceptance. The Discourse Quality Index seems to be the most sophisticated and promising of such methods, but there are still a number of problems that should be addressed. Read the entire thesis (PDF)

Does lack of rhetorical skills make you less deliberative?

Sverre — Fri, 20 Nov 2009 11:20:25 +0000

I’m working on a master thesis about political deliberation, and I’m interested in finding a good method for measuring and evaluating the level of deliberation in a discussion. Two current methods I’ve been looking into, the Discourse Quality Index and pragma-dialectics, both seem to share the same validity problem: They register low rhetorical or logical skills for a lack of deliberative attitude.

Put simply, deliberation is a term that is used about discussion when people engage in a rational dialogue about something in which they are dedicated to the “forceless force of the better argument”. According to Jürgen Habermas, its most famous theoretician, it should be characterized by sincerity, inclusiveness, equality, reasoned critique, reflexivity, respect and be free from the influence of money and coercive power¹ . In the more realistic conceptions of the term, we are willing to consider a discussion as more or less deliberative, realizing that a few real world discussions are likely to be perfect.

As I mentioned, I’ve been looking for methods to analyse discussion in order to determine how deliberative it really is. I’ve been trying to find a method that satisfies the criteria of:

Significance – Must be a method we can expect a large portion of empirically oriented political science to accept.
Usefulness – Must be a method that is suitable for comparative study of cases, preferably on a large scale and with a multitude of institutional arrangements.
Completeness – Must be a method that is theoretically consistent with established theory of deliberation, for example Habermas’ discourse ethics.

Lately I’ve been looking at two such methods: Pragma-dialectics² and the Discourse Quality Index³ . They both have their strength and weaknesses, but they share one interesting problem with validity.

In order to determine whether or not a debate is really deliberative, they both focus on the correctness of arguments presented. According to the DQI method, they evaluate the level of justification, classifying each speech act as either containing no justification (0), inferior justification (1), qualified justification (2) or sophisticated justification (3). The pragma-dialectical method does something similar when it examines entire debates for logical consistency, identifying so-called fallacies that constitute bad arguments which should not be part of reasoned discussion.

My claim is that both these methods measure something separate from the level of deliberation, they measure the level of sophistication of the debatants. They measure how good their rhetorical and logical skills are, not their dedication to reasoned debate and fair discussion. This is not a major problem when you compare debates where you can expect people to be at a similar skill level, as when comparing two parliaments. The problem arises when you want to compare the debates in parliament to debates in other areas of society. If you compare two groups with different levels of rhetorical skill, the less sophisticated group will register as less deliberative even if they really were more dedicated to the ideals of deliberation. The core principles of deliberation dictate that the best argument should prevail no matter who presents it, but if we use rhetorical sophistication as a measurement criterion, we are already violating it by derogating the arguments of the less educated.

This is still a work in progress, but it seems likely that this specific find will play a major part in the conclusions to my thesis.

My rewrite of the rules presented in his book discourse ethics
described most elegantly in a conference paper by Nicole Curato
Outlined in: Steiner, Jürg, André Bächtiger, Markus Spörndli and Marco R. Steenbergen (2004). Deliberative Politics in Action: Analyzing Parliamentary Discourse, Cambridge: Cambridge University Press.

How does Obama spend his time?

Sverre — Wed, 30 Sep 2009 11:26:50 +0000

Just came across POTUS Tracker, an interesting little tool from The Washington Post that lets you track what Obama emphasizes by how he spends his time in meetings. Apparently foreign policy and the economy are what he spends most of his time on, with health care only clocking in at place no. 5.

(Hat tip to Pravda for finding this).

Experiment on election prediction markets

Sverre — Wed, 05 Aug 2009 14:36:05 +0000

I’ve recently become involved (as a participant) in an interesting experiment performed by PhD student Sveinung Arnesen at the University of Bergen in which we are asked to predict the election result through a market model, buying and selling fictive “shares” in the outcome based on our own evaluations. This is based on prior experiments like Iowa Electronic Markets experiments by the University of Iowa in connection with American Presidential Elections, and the work of Robin Hanson.

Participants have been recruited through the political party organizations (at least I was), and appear to only have the option of buying or selling “stock” in our own party and/or government coalition. I assume part of the reason why we are restricted to our own party is the need for keeping the results secret to avoid incentives for strategic attempts at driving up the predicted value.

Unlike the Iowa experiments, this experiment is not conducted with “real money”. Each participant is given a number of virtual credits, and as far as I know there is no winning prize. This does of course give weak incentives towards predicting results accurately. If the result wasn’t being kept secret until election day, we might even have a serious adverse incentive to manipulate the predicted result to give our party “wind in the sails” towards the election. Such secrecy seems to be maintained, though, also to the extent of not letting us know the predictions made on other parties.

It seems highly likely to me, though, that the weak incentives for “winning” will cause party member participants to systematically overvalue their own party’s election result, especially when unable to also “invest” in other parties. The only alternative to holding “stock” in your own party is not holding stock…. i.e. having inert credits.

Of course it would possibly be likely that such a bias is so systematically distributed that it can be predicted and controlled for trhough discounting by some factor. Still I can’t help but think that there would be a chance for better predictions if we were either allowed to “invest” in all parties or had actual economic incentives to “win”.

In any case it’s interesting to see new methods of prediction getting tested. The Iowa results look rather good.

No Hawthorne effect at Hawthorne?

Sverre — Tue, 09 Jun 2009 21:13:55 +0000

Is nothing holy? Must I throw out my introduction textbook in methods now? The guys over at the Freakonomics blog have dug up and reexamined the original Hawthorne data and concluded that there actually was no Hawthorne effect in the original Hawthorne study! I’m disappointed right to the very core of my post-graduate student soul. What should i believe in now?

Hyperlinked data – Tim Berners’ vision for the new internet

Sverre — Fri, 20 Mar 2009 19:26:25 +0000

TED talks are very often great videos to watch, but this one should be a particular wet dream for every empirically oriented scientist:

[youtube=http://www.youtube.com/watch?v=OM6XIICm_qo]

Anti-naturalism – the truth about social science?

Sverre — Fri, 20 Mar 2009 10:33:31 +0000

Discussing the philosophy of science of the social sciences is always interesting, at least for those of us that are academically nerdy enough. LFC, the author of Howl at Pluto has highlighted the article “Concept Formation in Political Science: An Anti-Naturalist Critique of Qualitative Methodology” by Mark Bevir and Asaf Kedar in which the authors go against the naturalist focus on causal relationships in the social sciences. LFC’s analysis of their work is summed up in the following paragraph:

This all points to a more basic issue: Is there only one correct, philosophically defensible way to do social science? Some scholars believe that only an approach aimed at causal explanation is valid. B&K take the opposite side but adhere to an equivalent exclusiveness. The implication of their position seems quite clear: only one kind of social science will pass muster.

If I interpret LFC correctly, we both agree that both major philosophical ideas of social science has their merit and have contributed to social science as a whole. His post made me interested in reading the entire piece, which in a way surprised me and made me think even if I for the most part disagree with it.

On page 25, Bevir and Kedar state their mission as:

Before we turn to Sartori and Collier, however, we wish to reiterate that our critique of them is a philosophical one. Our critique attempts to unearth the philosophical assumptions in their methodology, showing them to be naturalistic and hence, given the foregoing arguments, inappropriate for political analysis. We seek thereby to shift the debate from the practical advantages of methodological strategies to their underlying philosophical assumptions. Given our philosophical agenda, there is no need for us to examine the soundness or quality of the substantive outcomes of Collier and Sartori’s approaches to concept formation. Rather, our critical task will have been fulfilled once we manage to demonstrate that those scholars’ methodologies are marked by a discredited naturalism.

They appear to be going on a crusade against current qualitative methodology on the grounds that it has accepted too much of the philosophy of science of naturalism and has thus undermined its own philosophical foundation. In their view, this makes current social science irrelevant.

In my opinion – and this is in large part what makes social science interesting – the acceptance of naturalist methodology, if not a strict naturalist philosophy of science, is a pragmatic way of accepting something less than complete knowledge as something that might still be useful. For me, the essence of social science is the ability to combine philosophical ideals and humanist concepts of understanding and intentionality with naturalistic conceptions of empiricism and causation to create better, if not complete, understanding of phenomena.

It appears to me that it is such perversions of the search for ultimate enlightenment that Bevir and Kedar are crusading against. And this is where I think they are moving beyond the realm of social science altogether.

The line they try to draw between themselves and the ‘naturalists’ (which is really more of a great gulf than a line) depends in larte part on the latter’s inattentiveness to the “holistic nature of meaning”. This concept – rather elusively defined in their article – appears to imply that intentionality is the only relevant aspect of human action, that study of outcomes is unimportant for understanding. They are opposed to social scientists reducing “meaning” to a variable in a larger picture. In their words (p. 30):

This atomization of concepts forecloses the possibility of holistic explanations that would open out on to the whole web of beliefs of social actors. Here too we thus find the naturalist elision of meaningfulness.

…and thus we get to their conclusion, and the essence of their work. I’m not familiar enough with Sartori and Collier to assess whether or not their analysis of these two authors make sense, so I stick with the general conclusions:

We began this essay by showing that there has arisen a widespread agreement (among philosophers if not in the unreflective practice of many social scientists) that anti-naturalist premises are the most appropriate for social science, where anti-naturalism highlights the meaningful and contingent nature of social life, the situatedness of the scholar, and so the dialogical nature of social science.

It still appears to me like what they are trying to do is discard social science as a whole, which is pragmatic by nature, and return to the realm of philosophy.

Still, their thoughts are interesting to read and discuss. I thank LFC for giving me (yet another) interesting few hours of distraction from my thesis work that will surely make me a better and more reflected social scientist, but not necessarily one with good grades to show for it.

NYT polling standards

Sverre — Tue, 17 Mar 2009 10:11:32 +0000

I recently found out that the NYT actually have published standards for what polls they are willing to publish. It doesn’t appear to be an entirely new thing (the article is from september 2008), but I found it to be an encouraging surprise. I wish more media were as quality conscious.

The standards document seems to be very basic, but still lay down some important ground rules for minimum requirements for the credibility of a poll. I’d like to see it go into some more detail also on what is required of the questions, but that would of course have to be less concrete and authorative.

10 points to the NYT for a good start.

A tip of the hat to Bård Vegard Solhjell for bringing this to my attention.

Pollster biases revealed

Sverre — Mon, 03 Nov 2008 10:25:32 +0000

The Monkey Cage led me to a paper by Len Adleman and Mark Schiling that compares the election polls made by American networks. They’ve compared them to the polls made by Gallup And Rasmussen, and show that the political inclination of the networks seem to influence the polls. Fox’s polls show a trend of predicting more to the right, and CBS/NYT more to the left. This is really interesting.

I agree with Andrew Gelman who comments that this surprises him. I would expect their coverage of the polls to show some bias, but had expected the polls themselves to be done in a professional way eliminating personal biases. This apparently goes to show that being completely neutral is difficult if not impossible, even in quantitative analysis.

What if the whole world could vote?

Sverre — Thu, 30 Oct 2008 10:32:22 +0000

… asks the Economist and tests it. They’ve asked their online readers to vote and constructed a worldwide electoral college. Lo and behold! the world electorate map is shockingly enough painted bright blue. It appears most nations in the world have a distribution in excess of 80-20 in Obama’s favour. Some people (no serious political scientists I hope) take this as evidence that the world supports Obama.

Economist.com voter map screenshot

The world at large probably prefers Obama, but this “survey” does in no way confirm that. Why not? It’s really quite simple. The survey is conducted at the Economist.com website. And who are going to claim that worldwide readers of the Economist represent a fair approximation to a random distribution of the population? None, I hope. For example, I would expect the Economists’ readers to have vastly higher than average levels of education. This is further accentuated by the fact that not every visitor to the website can vote, just registered Economist.com members. This ensures that even those casual visitors not normally reading the Economist are even less likely to vote.

There is no easier way to prove the bias of the survey than just looking at the scores for the US in this survey. In this survey the US supports Obama by 81-19.

This isn’t even very original. I have seen several such maps made on the basis of different surveys already, and must have read a dozen different online articles on it.

So has the Economist suddenly gone naive and stupid? Of course not. This was never intended as a survey by the Economist, so just don’t make the mistake of treating it like one. What the Economist wanted was more hits to their website, and more registered members on their website. I’d guess this got them hundreds, if not thousands. It worked on me. More seriously on the idea of global representative voting, The International Student Festival in Trondheim (ISFiT) did something like this properly when they had gathered students from around the world last year. They put together a world parliament with a system of representative voting based on world regions to decide on a wide range of global issues. Read more about this experiment at tgde.org.