Know-how is not interpretable so algorithms are not interpretable
by Sebastian Benthall
I happened upon Hildreth and Kimble’s “The duality of knowledge” (2002) earlier this morning while writing this and have found it thought-provoking through to lunch.
What’s interesting is that it is (a) 12 years old, (b) a rather straightforward analysis of information technology, expert systems, ‘knowledge management’, etc. in light of solid post-Enlightenment thinking about the nature of knowledge, and (c) an anticipation of the problems of ‘interpretability’ that were a couple months ago at least an active topic of academic discussion. Or so I hear.
This is the paper’s abstract:
Knowledge Management (KM) is a field that has attracted much attention both in academic and practitioner circles. Most KM projects appear to be primarily concerned with knowledge that can be quantified and can be captured, codified and stored – an approach more deserving of the label Information Management.
Recently there has been recognition that some knowledge cannot be quantified and cannot be captured, codified or stored. However, the predominant approach to the management of this knowledge remains to try to convert it to a form that can be handled using the ‘traditional’ approach.
In this paper, we argue that this approach is flawed and some knowledge simply cannot be captured. A method is needed which recognises that knowledge resides in people: not in machines or documents. We will argue that KM is essentially about people and the earlier technology driven approaches, which failed to consider this, were bound to be limited in their success. One possible way forward is offered by Communities of Practice, which provide an environment for people to develop knowledge through interaction with others in an environment where knowledge is created nurtured and sustained.
The authors point out that Knowledge Management (KM) is an extension of the earlier program of Artificial Intelligence, depends on a model of knowledge that maintains that knowledge can be explicitly represented and hence stored and transfered, and propose an alternative way of thinking about things based on the Communities of Practice framework.
A lot of their analysis is about the failures of “expert systems”, which is a term that has fallen out of use but means basically the same thing as the contemporary uncomputational scholarly use of ‘algorithm’. An expert system was a computer program designed to make decisions about things. Broadly speaking, a search engine is a kind of expert system. What’s changed are the particular techniques and algorithms that such systems employ, and their relationship with computing and sensing hardware.
Here’s what Hildreth and Kimble have to say about expert systems in 2002:
Viewing knowledge as a duality can help to explain the failure of some KM initiatives. When the harder aspects are abstracted in isolation the representation is incomplete: the softer aspects of knowledge must also be taken into account. Hargadon (1998) gives the example of a server holding past projects, but developers do not look there for solutions. As they put it, ‘the important knowledge is all in people’s heads’, that is the solutions on the server only represent the harder aspects of the knowledge. For a complete picture, the softer aspects are also necessary. Similarly, the expert systems of the 1980s can be seen as failing because they concentrated solely on the harder aspects of knowledge. Ignoring the softer aspects meant the picture was incomplete and the system could not be moved from the environment in which it was developed.
However, even knowledge that is ‘in people’s heads’ is not sufficient – the interactive aspect of Cook and Seely Brown’s (1999) ‘knowing’ must also be taken into account. This is one of the key aspects to the management of the softer side to knowledge.
In 2002, this kind of argument was seen as a valuable critique of artificial intelligence and the practices based on it as a paradigm. But already by 2002 this paradigm was falling away. Statistical computing, reinforcement learning, decision tree bagging, etc. were already in use at this time. These methods are “softer” in that they don’t require the “hard” concrete representations of the earlier artificial intelligence program, which I believe by that time was already refered to as “Good Old Fashioned AI” or GOFAI by a number of practicioners.
(I should note–that’s a term I learned while studying AI as an undergraduate in 2005.)
So throughout the 90’s and the 00’s, if not earlier, ‘AI’ transformed into ‘machine learning’ and become the implementation of ‘soft’ forms of knowledge. These systems are built to learn to perform a task optimally based flexibly on feedback from past performance. They are in fact the cybernetic systems imagined by Norbert Wiener.
Perplexing, then, is the contemporary problem that the models created by these machine learning algorithms are opaque to their creators. These models were created using techniques that were designed precisely to solve the problems that systems based on explicit, communicable knowledge were meant to solve.
If you accept the thesis that contemporary ‘algorithms’-driven systems are well-designed implementations of ‘soft’ knowledge systems, then you get some interesting conclusions.
First, forget about interpeting the learned models of these systems and testing them for things like social discrimination, which is apparently in vogue. The right place to focus attention is on the function being optimized. All these feedback-based systems–whether they be based on evolutionary algorithms, or convergence on local maxima, or reinforcement learning, or whatever–are designed to optimize some goal function. That goal function is the closest thing you will get to an explicit representation of the purpose of the algorithm. It may change over time, but it should be coded there explicitly.
Interestingly, this is exactly the sense of ‘purpose’ that Wiener proposed could be applied to physical systems in his landmark essay, published with Rosenbleuth and Bigelow, “Purpose, Behavior, and Teleology.” In 1943. Sly devil.
EDIT: An excellent analysis of how fairness can be represented as an explicit goal function can be found in Dwork et al. 2011.
Second, because what the algorithms is designed to optimize is generally going to be something like ‘maximize ad revenue’ and not anything particularly explicitly pernicious like ‘screw over the disadvantaged people’, this line of inquiry will raise some interesting questions about, for example, the relationship between capitalism and social justice. By “raise some interesting questions”, I mean, “reveal some uncomfortable truths everyone is already aware of”. Once it becomes clear that the whole discussion of “algorithms” and their inscrutability is just a way of talking about societal problems and entrenched political interests without talking about it, it will probably be tabled due to its political infeasibility.
That is (and I guess this is the third point) unless somebody can figure out how to explicitly define the social justice goals of the activists/advocates into a goal function that could be implemented by one of these soft-touch expert systems. That would be rad. Whether anybody would be interested in using or investing in such a system is an important open question. Not a wide open question–the answer is probably “Not really”–but just open enough to let some air onto the embers of my idealism.
As I posted in response to @bkeegan’s tweets of your entry (I’m an old lab-mate of his) – I’m unclear on a couple of things:
1. The term “contemporary uncomputational scholarly use of ‘algorithm’”. I haven’t read any contemporary communication papers since I left my Ph.D. program (6 years ago) to return to developing the software field to work on mobile versions of what you’re calling “expert systems”. What is/was lacking in the computational version of the term that required an uncomputational version for scholars?
2. I’d argue that there are plenty of non-digital “expert systems” that have been in use throughout history that have also had goal functions that ignored social justice concerns (prison sentencing guidelines), and it’s unclear about what makes interacting with and ultimately influencing an “algorithm” any different than those other systems.
Is the issue related to the opacity of the typical algorithmic system (Google’s search ranking specifics are proprietary) that prevents us from looking into the black box, of the sheer complexity of processes used by the algorithm that presents the problem?
If the first, what makes algorithms different than any sort of closed meeting held by humans (to make a decision), and if it’s the second, is the problem that the “interpretability” of algorithm is not distributed in a uniform or equitable manner?
For example, the designer and implementor of the algorithm *must* be able to interpret its decision making process, otherwise the system is not debuggable. I may not be able to make heads or tails from a collection of support vectors coming out of an SVM, but there is some form of logic (created and understandable by at least one human) that guides the creation of the learning system.
I guess the core question I’m asking is what is it about algorithms/expert systems (as executed by a machine) that make them different than any other sort of decision-making apparatus employed by humanity throughout history?
Apologies for the typos above:
“return to developing the software field” should have been “return to the software field”
“into the black box, of the sheer complexity” should have been “into the black box, or the sheer complexity”
I’ve been spoiled by edit features on other platforms.
(Also, keep up the good work!)
Thanks for these questions! Let me see if I can answer them.
1. This is confusing to me as well. The word ‘algorithm’ has a specific and banal meaning within computer science. What has happened is that since e.g. Google has become more socially relevant, scholars from non-computational fields need to discuss the impact of these systems in their own discourse. So they wind up talking about ‘algorithms’, but sort of as a stand in for a server with some installed software. It is bizarre to me; it seems to be a way for them to put the algorithm in a black box so they can discuss it without analyzing it in detail, or to discuss the social context around it.
Here’s an example of this kind of discourse. I wonder what you make of it?
To me this kind of discourse is very frustrating because it seems to operate by alternately conflating a bunch of things that aren’t the same and then disentangling them. I think your remaining questions are all valid confusions that this kind of discussion causes.
2. I agree that this distinction is strange, that there are human procedures that are basically algorithmic.
I think that proprietary software, complexity, and lack of training are three distinct sources of obscurity and that these get conflated. Each of these can and has been dealt with in plenty of literature on its own already. That’s why it’s so frustrating when these then get collapsed into a single point. Different kinds of ignorance look the same only to the unknowing.
So what makes algorithms different from other forms of decision-making?That’s a good question. I guess I would say the physical characteristics of how they are executed. They compute certain things very quickly. They are connected via wires and wireless transmission to sensors that give them a lot to work with. They are the product of rigorous intersubjective evaluation in ways that a lot of more primitive decision-making procedures are not.
What do you think about any of that?
First of all, apologies for the delay in my response. I was waiting for an e-mail ping when you responded, but failed to see that I needed to confirm my e-mail first. Doh!
Re: Gillespie’s talk – I honestly didn’t get much out of this given that Gillespie’s entire premise was based on responding to rhetoric about algorithms (largely originating from Google) rather than looking at the more interesting question of whether and how technologies like PageRank embody or enact certain values. Just because Google PR says that an algorithm is neutral doesn’t really carry a lot of weight (to me) and it seemed like a lot of time was spent responding to what proponents of the technology said about their tool rather than looking at the tool itself. His opening bit about Siri and abortion seemed to capture that frustration for me. The fact that Apple has decided to neuter Siri in some content areas when it comes to searches is somewhat interesting, but I don’t think it’s any more relevant or novel than auto manufacturers equipping combustion engines with speed governors.
With respect to the algorithms themselves as being different kinds of decision making tools, it woulds like we’re both trying to figure out if there’s actually anything new here or all of the talk of algorithms and algorithmic culture is rehashing old arguments like “do guns kill people or do people kill people?”.
In my own thoughts on the matter, I think that comm & technology scholars are focusing on the wrong angle when it comes to what makes an algorithm (in the “expanded” sense) an interesting object of inquiry. Now, I’m out of date with he STS literature (if I was ever “up to date” to begin with…), but a more useful example of how algs. are different in terms of decision making are probably trading algorithms or defense systems like SAGE:
http://en.wikipedia.org/wiki/Semi-Automatic_Ground_Environment
From the Wikipedia article:
“Later additions to the system allowed SAGE’s tracking data to be sent directly to CIM-10 Bomarc missiles and some of the US Air Force’s interceptor aircraft in-flight, directly updating their autopilots to maintain an intercept course without operator intervention.”
What SAGE and trading algorithms have in common are explicit removal of human judgement from John Boyd’s OODA loop in order to compensate for a human weakness (they’re too slow to complete the loop).
I may be agitating for an angle that I find a bit more relevant, but the reason I think that SAGE & HFT are better “algorithmic” examples because they probably illustrate “algorithmic nature” in a much purer form than someone at Google making a manual decision to remove Michelle Obama from their index. I feel like scholars (like Gillespie) who spend so much time talking about examples like companies tweaking their search results are not saying anything interesting or new and that they’re missing out on the much more important question of how fully automated decision making apparatuses are fundamentally different than processes with humans in the loop.
What interesting differences do we find when we compare an automated American defense network with a more manual Soviet system in the 1970’s. (Related – how did the Soviets’ approach to algorithmic defense – Dead Hand – differ from American’s SAGE in the 1980’s?)
Similarly, what are the effects of automated market activity (HFT) versus traditional manual trading? I feel like the differences are well understood in this case, but as someone building my own algorithmic tools – Purple Robot – I’m unclear what lessons I should be taking away from the comparison.
Hopefully my blathering is coherent and on-point here. Thanks once more for the video and your response.
Sabotaged once by autocorrect:
“it woulds like” should be “it sounds like”
LOL – autocorrect may be worth looking into all by itself.