Digifesto

Tag: qualitative research methods

functional determinism or overfitting to chaos

It’s been a long time since I read any Foucault.

The last time I tried, I believe the writing made me angry. He jumps around between anecdotes, draws spurious conclusions. At the time I was much sharper and more demanding and would not tolerate a fallacious logical inference.

It’s years later and I am softer and more flexible. I’m finding myself liking Foucault more, even compelled by his arguments. But I think I was just able to catch myself believing something I shouldn’t have, and needed to make a note.

Foucault brilliantly takes a complex phenomenon–like a prison and the society around it–and traces how its rhetoric, its social effects, etc. all reinforce each other. He describes a complex, and convinces the reader that the complex is a stable unit is society. Delinquency is not the failure of prison, it is the success of prison, because it is a useful category of illegality made possible by the prison. Etc.

I believe this qualifies as “rich qualitative analysis.” Qualitative work has lately been lauded for its “richness”, which is an interesting term. I’m thinking for example for the Human Centered Data Science CfP for CSCW 2016.

With this kind of work–is Foucault a historian? a theorist?–there is always the question of generalizability. What makes Foucault’s account of prisons compelling to me today is that it matches my conception of how prisons still work. I have heard a lot about prisons. I watched The Wire. I know about the cradle-to-prison system.

No doubt these narratives were partly inspired, enabled, by Foucault. I believe them, not having any particular expertise in crime, because I have absorbed an ideology that sees the systemic links between these social forces.

Here is my doubt: what if there are even more factors in play than have been captured by Foucault or a prevailing ideology of crime? What is prisons both, paradoxically, create delinquency and also reform criminals? What if social reality is not merely poststructural, but unstructured, and the narratives we bring to bear on it in order to understand it are rich because they leave out complexity, not because they bring more of it in?

Another example: the ubiquitous discourse on privilege and its systemic effect of reproducing inequality. We are told to believe in systems of privilege–whiteness, wealth, masculinity, and so on. I will confess: I am one of the Most Privileged Men, and so I can see how these forms of privilege reinforce each other (or not). But I can also see variations to this simplistic schema, alterations, exceptions.

And so I have my suspicions. Inequality is reproduced; we know this because the numbers (about income, for example), are distributed in bizarre proportions. 1% owns 99%! It must be because of systemic effects.

But we know now that many of the distributions we once believed were power law distributions created by generative processes such as preferential attachment are really log normal distributions, which are quite different. This is an empirically detectable difference whose implications are quite profound.

Why?

Because a log normal distribution is created not by any precise “rich get rich” dynamic, but rather by any process according to which random variables are multiplied together. As a result, you get extreme inequality in a distribution simply by virtue of how various random factors contributing towards it are mathematically combined (multiplicatively), as opposed to any precise determination of the factors upon each other.

The implication of this is that no particular reform is going to remove the skew from the distribution as long as people are not prevented from efficiently using their advantage–whatever it is–to get more advantage. Rather, reforms that are not on the extreme end (such as reparations or land reform) are unlikely to change the equity outcome except from the politically motivated perspective of an interest group.

I was pretty surprised when I figured this out! The implication is that a lot of things that look very socially structured are actually explained by basic mathematical principles. I’m not sure what the theoretical implications of this are but I think there’s going to be a chapter in my dissertation about it.

digital qualities: some meditations on methodology

Text is a kind of data that is both qualitative (interpretable for the qualities it conveys) and qualitative (characterized by certain amounts of certain abstract tokens arranged in a specific order).

Statistical learning techniques are able to extract qualitative distinctions from quantitative data, through clustering processes for example. Non-parametric statistical methods allow qualitative distinctions to be extracted from quantitative data without specifying particular structure or features up front.

Many cognitive scientists and computational neuroscientists believe that this is more or less how perception works. The neurons in our eyes (for example) provide a certain kind of data to downstream neurons, which activate according to quantifiable regularities in neuron activation. A qualitative difference that we perceive is due to a statistical aggregation of these inputs in the context of a prior, physically definite, field of neural connectivity.

A source of debate in the social sciences is the relationship between qualitative and quantitative research methods. As heirs to the methods of harder sciences whose success is indubitable, quantitative research is often assumed to be credible up to the profound limits of its method. A significant amount of ink has been spilled distinguishing qualitative research from quantitative research and justifying it in the face of skeptical quantitative types.

Qualitative researchers, as a rule, work with text. This is trivially true due to the fact that a limiting condition of qualitative research appears to be the creation of a document explicating the research conclusions. But if we are to believe several instructional manuals on qualitative research, then the work of an e.g. ethnographer involves jottings, field notes, interview transcripts, media transcripts, coding of notes, axial coding of notes, theoretical coding of notes, or, more broadly, the noting of narratives (often written down), the interpreting of text, a hermeneutic exposition of hermeneutic expositions ad infinitum down an endless semiotic staircase.

Computer assisted qualitative data analysis software passes the Wikipedia test for “does it exist”.

Data processed by computers is necessarily quantitative. Hence, qualitative data is necessarily quantitative. This is unsurprising, since so much qualitative data is text. (See above).

We might ask: what makes the work qualitative researchers do qualitative as opposed to quantitative, if the data they work with with quantitative? We could answer: it’s their conclusions that are qualitative.

But so are the conclusions of a quantitative researcher. A hypothesis is, generally speaking, a qualitative assessment, that is then operationalized into a prediction whose correspondence with data can be captured quantitatively through a statistical model. The statistical apparatus is meant to guide our expectations of the generalizability of results.

Maybe the qualitative researcher isn’t trying to get generalized results. Maybe they are just reporting a specific instance. Maybe generalizations are up to the individual interpreter. Maybe social scientific research can only apply and elaborate on an ideal type, tell a good story. All further insight is beyond the purview of the social sciences.

Hey, I don’t mean to be insensitive about this, but I’ve got two practical considerations: first, do you expect anyone to pay you for research that is literally ungeneralizable? That has no predictive or informative impact on the future? Second, if you believe that, aren’t you basically giving up all ground on social prediction to economists? Do you really want that?

Then there’s the mixed methods researcher. Or, the researcher who in principle admits that mixed methods are possible. Sure, the quantitative folks are cool. We’d just rather be interviewing people because we don’t like math.

That’s alright. Math isn’t for everybody. It would be nice if computers did it for us. (See above)

What some people say is: qualitative research generates hypotheses, quantitative research tests hypotheses.

Listen: that is totally buying into the hegemony of quantitative methods by relegating qualitative methods to an auxiliary role with no authority.

Let’s accept that hegemony as an assumption for a second, just to see where it goes. All authority comes from a quantitatively supported judgment. This includes the assessment of the qualitative researcher.

We might ask, “Where are the missing scientists?” about qualitative research, if it is to have any authority at all, even in its auxiliary role.

What would Bruno Latour do?

We could locate the missing scientists in the technological artifacts that qualitative researchers engage with. The missing scientists may lie within the computer assisted qualitative data analysis software, which dutifully treats qualitative data as numbers, and tests the data experimentally and in a controlled way. The user interface is the software’s experimental instrument, through which it elicits “qualitative” judgments from its users. Of course, to the software, the qualitative judgments are quantitative data about the cognitive systems of the software’s users, black boxes that nevertheless have a mysterious regularity to them. The better the coding of the qualitative data, the better the mysteries of the black box users are consolidated into regularities. From the perspective of the computer assisted qualitative data analysis software, the whole world, including its users, is quantitative. By delegating quantitative effort to this software, we conserve the total mass of science in the universe. The missing mass is in the software. Or, maybe, in the visual system of the qualitative researchers, which performs non-parametric statistical inference on the available sensory data as delivered by photo-transmitters in the eye.

I’m sorry. I have to stop. Did you enjoy that? Did my Latourian analysis convince you of the primacy or at least irreducibility of the quantitative element within the social sciences?

I have a confession. Everything I’ve ever read by Latour smells like bullshit to me. If writing that here and now means I will never be employed in a university, then may God have mercy on the soul of academe, because its mind is rotten and its body dissolute. He is obviously a brilliant man but as far as I can tell nothing he writes is true. That said, if you are inclined to disagree, I challenge you to refute my Latourian analysis above, else weep before the might of quantification, which will forever dominate the process of inquiry, if not in man, then in our robot overlords and the unconscious neurological processes that prefigure them.

This is all absurd, of course. Simultaneously accepting the hegemony of quantitative methods
and Latourian analysis has provided us with a reductio ad absurdum that compels us to negate some assumptions. If we discard Latourian analysis, then our quantitative “hegemony” dissolves as more and more quantitative work is performed by unthinking technology. All research becomes qualitative, a scholarly consideration of the poetry outputted by our software and instruments.

Nope, that’s not it either. Because somebody is building that software and those instruments, and that requires generalizability of knowledge, which so far qualitative methods have given up a precise claim to.

I’m going to skip some steps and cut to the chase:

I think the quantitative/qualitative distinction in social scientific research, and in research in general, is dumb.

I think researchers should recognize the fungibility of quantity and quality in text and other kinds of data. I think ethnographers and statistical learning theorists should warmly embrace each other and experience the bliss that is finding ones complement.

Goodnight.