Digifesto

So many projects!

I’m very happy with my current projects. Things are absolutely clicking. Here is some of what’s up.

I’ve been working alongside DARPA-recipients Dr. Liechty, Zak David, and Dr. Chris Carroll to develop SHARKFin, an open source system for Simulating Heterogeneous Agents with Rational Knowledge and the Financial system. SHARKFin builds on HARK, a system for macroeconomic modeling, but adds integration with complex ABM-based simulations of the financial system. The project straddles the gap between traditional macroeconomic theory and ABM-driven financial research in the style of Blake LeBaron and Richard Bookstaber. I’m so lucky to be part of the project; it’s a fascinating and important set of problems.

Particularly of interest is the challenge of reconciling the Rational Expectations assumption in economics — the idea that agents in a model know the model that they are in and act rationally within it — with the realities of the complexity of the financial system and the intractability that perhaps introduces into the model. The key question we seem to keep asking ourselves is: Is this publishable in an Economics Journal? Being perhaps too contrarian, I wonder: what does it mean for economics-driven public policy if intractable complexity is endogenous the the system? In a perhaps more speculative and ambitious project with Aniket Kesari, we are exploring inefficiencies in the data economy due to the problems with data market microstructure.

Because information asymmetries and bounded rationality increase this complexity, my core NSF research project, which was to develop a framework for heterogeneous agent modeling of the data economy, runs directly into this modeling tractability problems. Luckily for me, I’ve been attending meetings of the Causal Influence Working Group, which is working on many foundational issues in influence diagrams. This exposure has been most useful in helping me think through the design of multi-agent influence diagram models, which is my preferred modeling technique because of how it naturally handles situated information flows.

On the privacy research front, I’m working with Rachel Cummings on integrating Differential Privacy and Contextual Integrity. These two frameworks are like peanut butter and jelly — quite unlike each other, and better together. We’ve gotten a good reception for these ideas at PLSC ’22 and PEPR ’22, and will be presenting a poster about it this week at TPDP ’22. I think influence diagrams are going to help us with this integration as well!

Meanwhile, I have an ongoing project with David Shekman wherein we are surveying the legal and technical foundations for fiduciary duties for computational systems. I’ve come to see this as the right intersection between Contextual Integrity and aligned data policy initiatives and the AI Safety research agenda, specifically AI Alignment problems. While often considered a different disciplinary domain, I see this problem as the flip side of the problems that come up in the general data economy problem. I expect the results, once they start manifesting, to spill over onto each other.

With my co-PIs we are exploring the use of ABMs for software accountability. The key idea here is that computational verification of software accountability requires a model of the system’s dynamic environment — so why not build the model and test the consequences of the software in silico? So far in this project we have used classic ABM models which do not require training agents, but you could see how the problem expands and overlaps with the economic modeling issues raised above. But this project makes use confront quite directly the basic questions of simulations as a method: how can they be calibrated or validated? When and how should they be relied on for consequential policy decisions?

For fun, I have joined the American Society for Cybernetics, which has recently started a new mailing list for “conversations”. It’s hard to overstate how fascinating cybernetics is as kind of mirror phenomenon to contemporary AI, computer science, and economics. Randy Whitaker, who I’m convinced is the world’s leading expert on the work of Humberto Maturana, is single-handedly worth the price of admission to the mailing list, which is the membership fee of ASC. If you have any curiosity about the work of Maturana and Varela and their ‘biology of cognition’ work, this community is happy to discuss its contextual roots. Many members of ASC knew Maturana and Francisco Varela personally, not to mention others like Gregory Bateson and Heinz von Foerster. My curiosity about ‘what happened to cybernetics?’ has been, perhaps, sated — I hope to write a little about what I’ve learned at some point. Folks at ASC, of course, insist that cybernetics will have its come-back any day now. Very helpfully, through my conversations at ASC I’ve managed to convince myself that many of the more subtle philosophical or psychological questions I’ve had can in fact be modeled using modified versions of the Markov Decision Process framework and other rational agent models, and that there are some very juicy results lying in wait there if I could find the time to write them up.

I’m working hard but feel like at last I’m really making progress on some key problems that have motivated me for a long time. Transitioning to work on computational social simulations a few years ago has scratched an itch that was bothering me all through my graduate school training: mere data science, with its shallowly atheoretic and rigidly empirical approach, to me misses the point on so many research problems, where endogenous causal effects, systemic social structure, and sociotechnical organization are the phenomena of interest. Luckily, the computer science and AI communities seem to be opening up interest in just this kind of modeling, and the general science venues have long supported this line of work. So at last I believe I’ve found my research niche. I just need to keep funded so that these projects can come to fruition!

Buried somewhere in this work are ideas for a product or even a company, and I dream sometimes of building something organizational around this work. A delight of open source software as a research method is that technology transfer is relatively easy. We are hopeful that SHARKFin will have some uptake at a government agency, for example. HARK is still in early stages but I think has the potential to evolve into a very powerful framework for modeling multi-agent systems and stochastic dynamic control, an area of AI that is currently overshadowed by Deep Everything but which I think has great potential in many applications.

Things are bright. My only misgiving is that it took my so long to find and embark on these research problems and methods. I’m impatient with myself, as these are all deep fields with plenty of hardworking experts and specialists that have been doing it for much longer than I have. Luckily I have strong and friendly collaborators who seem to think I have something to offer. It is wonderful to be doing such good work.

Leave a comment

February 4, 2022

About ethics and families

Most of the great historical philosophers did not have children.

I can understand why. For much of my life, I’ve been propelled by a desire to understand certain theoretical fundamentals of knowledge, ethics, and the universe. No doubt this has led me to become the scientist I am today. Since becoming a father, I have less time for these questions. I find myself involved in more mundane details of life, and find myself beginning to envy those in what I had previously considered the most banal professions. Fatherhood involves a practical responsibility that comes front-and-center, displacing youthful ideals and speculations.

I’m quite proud to now be working on what are for me rather applied problems. But these problems have deep philosophical roots and I enjoy the thought that I will one day be able to write a mature philosophy as a much older man some time later. For now, I would like to jot down a few notes about how my philosophy has changed.

I write this now because my work is now intersecting with other research done by folks I know are profoundly ethically motivated people. My work on what is prosaically called “technology policy” is crossing into theoretical territory currently occupied by AI Safety researchers of the rationalist or Effective Altruist vein. I’ve encountered these folks before and respect their philosophical rigor, though I’ve never quite found myself in agreement with them. I continue to work on problems in legal theory as well, which always involves straddling the gap between consequentialism and deontological ethics. My more critical colleagues may be skeptical of my move towards quantitative economic methods, as the latter are associated with a politics that has been accused of lacking integrity. In short, I have several reasons to want to explain, to myself at least, why I’m working on the problems I’ve chosen, at least as a matter of my own philosophical trajectory.

So first, a point about logic. The principle of non-contradiction imposes a certain consistency and rigor on thought and encourages a form of universalism of theory and ethics. The internal consistency of the Kantian transcendental subject is the first foundation for deontological ethics. However, for what are essentially limitations of bounded rationality, this gives way in later theory to Habermasian discourse ethics. The internal consistency of the mind is replaced with the condition that to be involved in communicative action is to strive for agreement. Norms form from disinterested communications that collect and transcend the perspectival limits of the deliberators. In theory.

In practice, disinterested communication is all but impossible, and communicative competence is hard to find. At the time of this writing, my son does not yet know how to talk. But he communicates, and we do settle on norms, however transitory. The other day we established that he is not allowed to remove dirt from the big pot with the ficus elastica and deposit in other rooms of the house. This is a small accomplishment, but it highlights how unequal rationality, competence, and authority is not a secondary social aberration. It is a primary condition of life.

So much for deontology. Consequential ethics does not fare much better. Utility has always been a weakly theorized construct. In modern theory, it has been mathematized into something substantively meaningless. It serves mainly to describe behavior, rather than to explain it; it provides little except a just-so-story for a consumerist society which is, sure enough, best at consuming itself. Attempts to link utility to something like psychological pleasure, as was done in the olden days, have bizarre conclusions. Parents are not as happy, studies say, as those without children. So why bother?

Nietzsche was a fierce critic of both Kantian deontological ethics and facile British utilitarianism. He argued that in the face of the absurdity of both systems, the philosopher had to derive new values from the one principle that they could not, logically, deny: life itself. He believed that a new ethics could be derived from the conditions of life, which for him was a process of overcoming resistance in pursuit of other (perhaps arbitrary) goals. Suffering, for Nietzsche, was not a blemish on life; rather, life is sacred enough to justify monstrous amounts of suffering.

Nietzsche went insane and died before he could finish his moral project. He didn’t have kids. If he had, maybe he would have come to some new conclusions about the basis for ethics.

In my humble opinion and limited experience thus far, fatherhood is largely about working to maintain the conditions of life for one’s family. Any attempt at universalism that does not extend to one’s own offspring is a practical contradiction when one considers how one was once a child. The biological chain of being is direct, immediate, and resource intensive in a way too little acknowledged in philosophical theory.

In lieu of individual utility, the reality of family highlights the priority of viability, or the capacity of a complex, living system to maintain itself and its autonomy over time. The theory of viability was developed in the 20th century through the field of cybernetics — for example, by Stafford Beer — though it was never quite successfully formulated or integrated into the now hegemonic STEM disciplines. Nevertheless, viability provides a scientific criterion by which to evaluate social meaning and ethics. I believe that there is still tremendous potential in cybernetics as an answer to longstanding philosophical quandaries, though to truly capture this value certain mathematical claims need to be fleshed out.

However, an admission of the biological connection between human beings cannot eclipse economic realities that, like it or not, have structured human life for thousands of years. And indeed, in these early days of child-rearing, I find myself ill-equipped to address all of my son’s biological needs relative to my wife and instead have a comparative advantage in the economic aspects of his, our, lives. And so my current work, which involves computational macroeconomics and the governance of technology, is in fact profoundly personal and of essential ethical importance. Economics has a reputation today for being a technical and politically compromised discipline. We forget that it was originally, and maybe still is, a branch of moral philosophy deeply engaged with questions of justice precisely because it addresses the conditions of life. This ethical imperative persists despite, or indeed because of, its technical complexity. It may be where STEM can address questions of ethics directly. If only it had the right tools.

In summary, I see promise in the possibility of computational economics, if inspired by some currently marginalized ideas from cybernetics, in satisfactorily addressing some perplexing philosophical questions. My thirsting curiosity, at the very least, is slaked by daily progress along this path. I find in it the mathematical rigor I require. At the same time, there is space in this work for grappling with the troublingly political, including the politics of gender and race, which are both of course inexorably tangled with the reality of families. What does it mean, for the politics of knowledge, if the central philosophical unit and subject of knowledge is not the individual, or the state, or the market, but the family? I have not encountered even the beginning of an answer in all my years of study.

Leave a comment

October 8, 2021

Towards a Synthesis of Differential Privacy and Contextual Integrity

At last week’s 3rd Annual Symposium on Applications of Contextual Integrity, there was a lively discussion of a top-of-mind concern for computers scientists seeking to work with Contextual Integrity (CI): how does CI relate to differential privacy (DP)? Yan Shvartzshnaider encouraged me to write up my own comments as a blog post.

Differential Privacy (DP)

Differential privacy (Dwork, 2006) is a widely studied paradigm of computational privacy. It is a mathematical property of an algorithm or database $A$ which dictates that the output of the mechanism depends only slightly on any one individual data subject’s data. This is most often expressed mathematically as

$Pr[A(D_1) \in S] \leq e^\epsilon \cdot Pr[A(D_2) \in S]$

Where $D_1$ and $D_2$ differ only by the contents on one data point corresponding to a single individual, and $S$ is any arbitrary set of outputs of the mechanism.

A key motivation for DP is that each individual should, in principle, be indifferent to whether or not they are included in the DP database, because their impact on the result is bounded by a small value, $\epsilon$ .

There are many, many variations of DP that differ based on assumptions about the generative model of the data set, the privacy threat model, and others ways of relaxing the indifference constraint. However, the technical research of DP is often silent on some key implementation details, such as how to choose the privacy budget $\epsilon$ . There are some noteworthy industrial applications of DP, but they may use egregiously high values of $\epsilon$ . There are also several reasons to believe the DP is not getting at a socially meaningful sense of privacy, but rather is merely a computationally convenient thing to research and implement.

Contextual Integrity (CI)

Contextual Integrity (Nissenbaum, 2009/2020) aims to capture what is socially meaningful about privacy. It defined privacy as appropriate information flow, where appropriateness means alignment with norms based in social context. Following Walzer (2008)’s vision of society divided into separate social spheres, CI recognizes that society is differentiated into many contexts, such as education, healthcare, the workplace, and the family, and that each context has different norms about personal information flow that are adapted to that context’s purpose. For example, the broadly understood rules that doctors keep their patient’s medical information confidential, but can share records with patient’s consent to other medical specialists, are examples of information norms that adhere in the context of healthcare. CI provides a template for understanding information norms, parameterized in terms of:

Sender of the personal information
Receiver of the personal information
Subject of the personal information — the “data subject” in legal terms
The attribute of the data subject that is referred to or described in the personal information.
The transmission principle, the normative rule governing the conditions under which the above parameterized information flow is (in)appropriate. Examples of transmission principles include reciprocity, confidentiality, and consent.

Though CI is a theory based in social, philosophical, and legal theories of privacy, it has had uptake in other disciplines, including computer science. These computer science applications have engaged CI deeply and contributed to it by clarifying the terms and limits of the theory (Benthall et al., 2017).

CI has perhaps been best used by computer scientists thus far as a way of conceptualizing the privacy rules of sectoral regulations such as HIPAA, GLBA, and COPPA (Barth et al., 2006) and commercial privacy polices (Shvartzshnaider et al., 2019). However, a promise of CI is that is can address social expectations that have not yet been codified into legal language, helping to bridge between technical design, social expectation, and legal regulation in new and emerging contexts.

Bridging Between DP and CI

I believe it’s safe to say that whereas DP has been widely understood and implemented by computer scientists, it has not sufficed as either a theory or practice to meet the complex and nuanced requirements that socially meaningful privacy entails. On the other hand, while CI does a better job of capturing socially meaningful privacy, it has not yet been computationally operationalized in a way that makes it amenable to widespread implementation. The interest at the Symposium in bridging DP and CI was due to a recognition that CI has defined problems worth solving by privacy oriented computer scientists who would like to build on their deep expertise in DP.

What, then, are the challenges to be addressed by a synthesis of DP and CI? These are just a few conjectures.

Social choice of epsilon. DP is a mathematical theory that leaves open the key question of the choice of privacy budget $\epsilon$ . DP researchers would love a socially well-grounded way to choose is numerical value. CI can theoretically provide that social expectation, except for the fact that social norms are generally not expressed with such mathematical sensitivity. Rather, social norms (and legal rules) use less granular terms like confidentiality and consent. A DP/CI synthesis might involve a mapping from natural language privacy rules to numerical values for tuning DP.

Being explicit about context. DP is attractive precisely because it is a property of a mechanism that does not depend on the system’s context (Tschantz et al., 2020). But this is also its weakness. Key assumptions behind the motivation of DP, such as that the data subjects’ qualities are independent from each other, are wrong in many important privacy contexts. Variations of DP have been developed to, for example, adapt to how genetically or socially related people will have similar data, but the choice of which variant to use should be tailored to the conditions of social context. CI can inform DP practice by clarifying which contextual conditions matter and how to map these to DP variations.

DP may only address a subset of CI’s transmission principles. The rather open concept of transmission principle in CI does a lot of work for the theory by making it extensible to almost any conceivable privacy norm. Computer scientists may need to accept that DP will only be able to address a subset of CI’s transmission principles — those related to negative rules of personal information flow. Indeed, some have argued that CI’s transmission principles include rules that will always be incompletely auditable from a computer science perspective. (Datta et al., 2001) DP scholars may need to accept the limits of DP and see CI as a new frontier.

Horizontal data relations and DP for data governance. Increasingly, legal privacy scholars are becoming skeptical that socially meaningful privacy can be guaranteed to individuals alone. Because any individual’s data can enable an inference that has an effect on others, even those who are not in the data set, privacy may not properly be an individual concern. Rather, as Viljoen argues, these horizontal relationships between individuals via their data make personal data a democratic concern properly addressed with a broader understanding of collective or institutional data governance. This democratic data approach is quite consistent with CI, which was among the first privacy theories to emphasize the importance of socially understood norms as opposed to privacy as individual “control” of data. DP can no longer rely on its motivating idea that individual indifference to inclusion in a data set is sufficient for normative, socially meaningful privacy. However, some DP scholars have already begun to expand their expertise and address how DP can play a role in data governance. (Zhang et al., 2020)

DP and CI are two significant lines of privacy research that have not yet been synthesized effectively. That presents an opportunity for researchers in either subfield to reach across the aisle and build new theoretical and computational tools for socially meaningful privacy. In many ways, CI has worked to understand the socially contextual aspects of privacy, preparing the way for more mathematically oriented DP scholars to operationalize them. However, DP scholars may need to relax some of their assumptions and open their minds to make the most of what CI has to offer computational privacy.

References

Barth, A., Datta, A., Mitchell, J. C., & Nissenbaum, H. (2006, May). Privacy and contextual integrity: Framework and applications. In 2006 IEEE symposium on security and privacy (S&P’06) (pp. 15-pp). IEEE.

Benthall, S., Gürses, S., & Nissenbaum, H. (2017). Contextual integrity through the lens of computer science. Now Publishers.

Datta, A., Blocki, J., Christin, N., DeYoung, H., Garg, D., Jia, L., Kaynar, D. and Sinha, A., 2011, December. Understanding and protecting privacy: Formal semantics and principled audit mechanisms. In International Conference on Information Systems Security (pp. 1-27). Springer, Berlin, Heidelberg.

Dwork, C. (2006, July). Differential privacy. In International Colloquium on Automata, Languages, and Programming (pp. 1-12). Springer, Berlin, Heidelberg.

Nissenbaum, H. (2020). Privacy in context. Stanford University Press.

Shvartzshnaider, Y., Pavlinovic, Z., Balashankar, A., Wies, T., Subramanian, L., Nissenbaum, H., & Mittal, P. (2019, May). Vaccine: Using contextual integrity for data leakage detection. In The World Wide Web Conference (pp. 1702-1712).

Shvartzshnaider, Y., Apthorpe, N., Feamster, N., & Nissenbaum, H. (2019, October). Going against the (appropriate) flow: A contextual integrity approach to privacy policy analysis. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing (Vol. 7, No. 1, pp. 162-170).

Tschantz, M. C., Sen, S., & Datta, A. (2020, May). Sok: Differential privacy as a causal property. In 2020 IEEE Symposium on Security and Privacy (SP) (pp. 354-371). IEEE.

Viljoen, S. (forthcoming). Democratic data: A relational theory for data governance. Yale Law Journal.

Walzer, M. (2008). Spheres of justice: A defense of pluralism and equality. Basic books.

Zhang, W., Ohrimenko, O., & Cummings, R. (2020). Attribute Privacy: Framework and Mechanisms. arXiv preprint arXiv:2009.04013.

Leave a comment

August 22, 2021

Research update: to study the economy of personal data

I have not been writing here for some time because of strokes of good luck that have been keeping me busy.

I’ve been awarded a Social Behavioral and Economic Sciences (SBE) Post-Doctoral Research Fellowship (“SPRF” in total) by the National Science Foundation.

This is a lot of words to write out, but they sum up to a significant change in my research and role that I’m still adjusting to.

First, I believe this means that I am a social scientist of some kind. What kind? It’s not clear. If I could have my choice, it would be “economist”. But since Economics is a field widely known for gatekeeping, and I do not have an Economic degree, I’m not sure I can get away with this.

Nevertheless, my SPRF research project is an investigation into the economics of data (especially personal data) using methods that are build on those used in orthodox and heterodox economics.

The study of the economics of personal data is coming from my dissertation work and the ongoing policy research I’ve done at NYU School of Law’s Information Law Institute. Though my work has touched on many other fields — computer science and the design of information systems; sociology and the study of race and networked publics; philosophy and law — at the end of the day the drivers of “technology’s” impact on society are businesses operating according to an economic logic. This is something that everybody knows, but that few academic researchers are in a position to admit, because many of the scholars who think seriously about these issues are coming from other disciplines.

For better or for worse, I have trouble sticking to a tunnel of which I can’t see the intellectual daylight at the end.

So how can we study the economy of personal data?

I would argue — that this is something that most Economists will balk at — that the tools currently available to study this economy are insufficient for the task. Who am I to say such a thing? Nobody special.

But weighing in my favor is the argument that the even the tools used by Economists to study the macroeconomy are insufficient for the task. This point was made decisively by the 2008 Financial Crisis, which blindsided the economic establishment. One of the reasons why Economics failed was because the discipline had deeply entrenched oversimplified assumptions in their economic models. One of these was representative agent modeling, which presumed to model the enter economy with a single “representative agent” for a sector or domain. This makes the economist’s calculations easier but is clearly unrealistic, and indeed it’s the differences between agents that create much of the dynamism and pitfalls of the economy. Hence the rise in heterogeneous agent modeling (HAM), which is explicit about the differences between agents with respect to things like, for example, wealth, risk aversion, discount factor, level of education, and so on.

It was my extraordinary good fortune to find an entry into the world of HAM via the Econ-ARK software project (Carroll et al, 2018; Benthall and Seth, 2020), which needed a software engineer enthusiastic about open source scientific tools at a moment when I was searching for a job. Econ-ARK’s HAM toolkit, HARK, has come a long way since I joined the project in late 2019. And it still has quite a ways to go. But it’s been a tremendously rewarding project to be involved with, in no small part because it has been a hands-on introduction to the nitty-gritty of contemporary Economics methods.

It’s these tools which I will be extending with insights from my other work, which is grounded more in computer science and legal scholarship, in order to model the data economy. Naturally, the economy for personal data depends on the heterogeneity of consumers — it is those differences that make a difference between consumers that make the trade in personal information possible and relevant. And while there are many notational and conventional differences between the orthodox Economics methods and the causal Bayesian frameworks that I’ve worked in before, these methods in fact share a logical core that makes them commensurable.

I’ve mentioned both orthodox and heterodox economics. By this I mean to draw a distinction between the core of the Economics discipline, which in my understanding is still tied to rational expectations and general equilibria — meaning the idea that agents know what to expect from the market and act accordingly — and heterodox views that find these assumptions to be dangerously unrealistic. This is truly a sore spot for Economics. As the trenchant critiques of Mirowski and Nik-Kah (2017) reveal, these core assumptions commit Economists to many absurd conclusions; however, they are loathe to abandon them lest they lose the tight form of rigor which they have demanding to maintain a kind of standardization within the discipline. Rational expectations aligns economics with engineering disciplines, like control theory and artificial intelligence, which makes their methods more in-demand. Equilibrium theories give Economics a normative force and excuses when its predictions do not pan out. However, the 2008 Financial Crisis embarassed these methods, and now the emerging HAM techniqes include not only a broadened from of rational agent modeling, but also a much looser paradigm of Agent-Based Modeling (ABM) that allow for more realistic dynamics with boundedly rational agents (Bookstaber, 2017).

Today, the biggest forces in the economy are precisely those that have marshaled information to their advantage in a world with heterogeneous agents (Benthall and Goldenfein, 2021). Economic agents differ both horizontally — like consumers of different demographic categories such as race and sex — and vertically — as consumers and producers of information services have different relationships to personal data. As I explore in forthcoming work with Salome Viljoen (2021), the monetization of personal data has always been tied to the financial system, first via credit reporting, and later through the financialization of consumer behavior through digital advertising networks. And yet the macroeconomic impact of the industries that profit from these information flows, which now account for the largest global companies, is not understood because of disciplinary blinders that Economics has had for decades and is only now trying to shed.

I’m convinced the research is well motivated. The objection, which comes from my most well-meaning mentors, is that the work is too difficult or in fact impossible. Introducing heterogeneously bounded rationality into economic modeling creates a great deal of modeling and computational complexity. Calibrating, simulating, and testing such models is expensive, and progress requires a great deal of technical thinking about how to compute results efficiently. There are also many social and disciplinary obstacles to this kind of work: for the reasons discussed above, it’s not clear where this work belongs.

However, I consider myself immensely fortunate to have a real, substantive, difficult problem to work on, and enough confidence from the National Science Foundation that they support my trying to solve it. It’s an opportunity of a lifetime and, to be honest, as a researcher who has often felt at the fringes of a viable scholarly career, a real break. The next steps are exciting and I can’t wait to see what’s around the corner.

References

Benthall, S., & Goldenfein, J. (2021, May). Artificial Intelligence and the Purpose of Social Systems. In Proceedings of the 2021 AAAI/ACM Conference on AI Ethics and Society (AIES’21).

Benthall, S., & Seth, M. (2020). Software Engineering as Research Method: Aligning Roles in Econ-ARK.

Benthall, S. & Viljoen, S. (2021) Data Market Discipline: From Financial Regulation to Data Governance. J. Int’l & Comparative Law https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3774418

Bookstaber, R. (2017). The end of theory. Princeton University Press.

Carroll, C. D., Kaufman, A. M., Kazil, J. L., Palmer, N. M., & White, M. N. (2018). The Econ-ARK and HARK: Open Source Tools for Computational Economics. In Proceedings of the 17th Python in Science Conference (pp. 25-30).

Mirowski, P., & Nik-Khah, E. (2017). The knowledge we have lost in information: the history of information in modern economics. Oxford University Press.

Leave a comment

March 15, 2021

Luhmann, Social Systems, Cha. 1 § I

Niklas Luhmann (1927-1998) was a German sociologist who aimed to understand society in terms of systems theory.

I am reading Luhmann’s Social Systems (1995) because I have a hunch that this theory is relevant to my research. This post contains notes about Chapter 1, section I-II.

Often, scientists need to sacrifice intelligibility for accuracy. Luhmann is a scientist. He is unapologetic about this. He opens his “Instead of a Preface to the English Edition” (actual title) with:

“This is not an easy book. It does not accommodate those who prefer a quick and easy read, yet do not want to die without a taste of systems theory. This holds for the German text, too. If one seriously undertakes to work out a comprehensive theory of the social and strives for sufficient conceptual precision, abstraction and complexity in the conceptual infrastructure are unavoidable.”

Why bother reading such a difficult book? Why be a scientist and study social systems?

One reason to study society scientifically is to design and build better smart digital infrastructure.

Most people designing and building smart digital infrastructure today are not studying Luhmann. They are studying computer science. That makes sense: computer science is a science of smart digital artifacts. What has become increasingly apparent in recent years is that smart digital infrastructure is having an impact on society, and that the infrastructure is often mismatched to its social context. These mismatches are often considered to be a problem. Hence, a science of society might inform better technical designs.

§I

Chapter 1 opens with:

The following considerations assume that there are systems. Thus they do not begin with epistemological doubt. They also do not advocate a “purely analytical relevance” for systems theory. The most narrow interpretation of systems theory as a mere method of analyzing reality is deliberately avoided. Of course, one must never confuse statements with their objects; one must realize that statements are only statements and that scientific statements are only scientific statements. But, as least in systems theory, they refer to the real world. Thus the concept of system refers to something that is in reality a system and thereby incurs the responsibility of testing its statements against reality.

This is a great opening. It is highly uncommon for work in the social sciences to begin this way. Today, social science is almost always taught in theoretically pluralistic way. The student is taught several different theories of the same phenomenon. As they specialize into a social scientific discipline, they are taught to reproduce that discipline by citing its canonical thinkers and apply its analytical tools to whatever new phenomenon presents itself.

Not so with Luhmann. Luhmann is trying to start from a general scientific theory — systems theory — that in principle applies to physical, biological, and other systems, and to apply it to social systems. He cites Talcott Parsons, but also Herbert Simon, Ludwig von Bertalanffy, and Humberto Maturana. Luhmnn is not interested in reproducing a social scientific field; he is interested in reproducing the scientific field of systems theory in the domain of social science.

So the book is going to:

Be about systems theory in general
Address how social systems are a kind of system
Address how social systems relate to other kinds of system

There is a major challenge to studying this book in 2021. That challenge is that “systems theory” is not a mainstream scientific field today, and that people that do talk about “systems” normally do so in the context of “systems engineering”, to study and design industrial processes for example. They have their own quantitative discipline and methodologies that has little to do with sociology. Computer scientists, meanwhile, will talk about software systems and information systems, but normally in a way that has nothing to do with “systems theory” or systems engineering in a mechanical sense. Hazarding a guess, I would say that this has something to do with the cybernetics/AI split in the second half of the 20th century.

There is now a great deal of convergence in mathematical notation and concepts between different STEM fields, in part because much of the computational tooling has become ubiquitous. Computational social science has made great strides in recent years as a result. But many computational social science studies apply machine learning techniques to data generated by a social process, despite the fact that nobody believes the model spaces used in machine learning contain a veridical model of society.

This has led to many of the ethical and social problems with “AI”. Just for brief example, it is well known that estimating fitness via regression for employment or parole from personal information is, even when sensitive categories are excluded, likely to reproduce existing societal biases extant in the data through proxy variables in the feature set. A more subtle causal analysis can perhaps do better, but the way causality works at a societal level is not straightforward. See Lily Hu’s discussion of this topic, for some deeper analysis. Understanding the possible causal structures of society, including the possibility of “bottom-up” emergent effects and “downward causation” effects from social structures, would potentially improve the process of infrastructure design, whether manual or automated (via machine learning).

With this motive in mind, we will continue to slowly analyze and distill Luhmann in search for relevant insights.

For Luhmann, “systems theory … claims universal validity for everything that is a system.” Implicitly, systems theory has perfect internal validity. Luhmann expresses this theory in German, originally. But it really feels like there should be a mathematization of this work. He does not cite one yet, but the spoiler is that he’s eventually going to use George Spencer-Brown’s Laws of Form. For reasons I may get into later if I continue with this project, I believe that’s an unfortunate choice. I may have to find a different way to do the mathematization.

Rather, he follows through on his commitment to the existence of real systems by inferring some necessary consequences of that first principle. He is not content with a mathematical representation; systems theory must have “a real reference to the world”; “it is forced to treat itself as one of its objects in order to compare itself with others among those objects”. The crux is that systems theory, being a system itself, has to be able to take itself into account from the start. Hence, the commitment to real systems entails the realness of self-referential systems. “This means … there are systems that have the ability to establish relations with themselves and to differentiate these relations from relations with their environment.”

We are still in §I, which is itself a sort of preamble situating systems theory as a scientific theory, but already Luhmann is exposing the substance of the theory; in doing so, he demonstrates how truly self-referential — and consistently so — systems theory is. As he’ll say more definitively later, one essential feature of a system is that it is different from its environment. A system has, in effect, an “inside”, and also an “outside”. Outside the system is the environment. The part of the system that separates the inside of the system and its environment is the boundary. This binary aspect of the system (the system, and the not-the-system (environment)) clarifies the logic of ‘self-reference’. Self-referential systems differentiate between themselves and not-themselves.

So far, you have perhaps noted that Luhmann is a terribly literal writing. It is no surprise that the focus of his book, Social Systems, is that subset of systems that are “social”. What are these systems like? What makes them different from organisms (also systems), or systems of machines? Luhmann eschews metaphor — a bold choice. “[W]e do not choose the shortcut of analogy, but rather the longer path of generalization and respecification.” We don’t want to be misled by analogies.

“Above all, we will have to emphasize the nonpsychic character of social systems.”

That’s something Luhmann says right after saying he doesn’t want to use metaphors when talking about social systems. What can this possibly mean? It means, among other things, that Luhmann is not interested in anybody’s subjective experience of a society as an account of what a social system is. A “psychic system”, like my lived experience, or yours, is not the same thing as the social system — though, as we will later read, psychic systems are “structurally coupled” with the social system in important ways. Rather, the social system is constituted, objectively, by the communications between people. This makes it a more ready object of science.

It is striking to me that Luhmann is not more popular among analysts of social media data, because at least superficially he seems to be arguing, in effect, the social system of Twitter is not the system of Twitter’s users. Rather, it’s the system of the tweets. That’s one way of looking at things, for sure. Somewhat abashedly, I will say that Luhmann is an interesting lens through which to view Weird Twitter, which you may recall as a joke-telling subculture of Twitter that was popular before Former President Trump made Twitter much, much weirder. I think there’s some interesting comparisons to be drawn between Anthony Cohen’s theory of the symbolic construction of community, complete with symbolic boundary, and Luhmann’s notion of the boundary of a social system. But I digress.

Luhmann hasn’t actually used the word “communication” yet. He instead says “social contact”. “Every social contact is understood as a system, up to and including society as the inclusion of all possible contacts.” Possible contacts. Meaning that the system is defined in part by its unrealized but potential states. It can be stochastic; it can be changing its internal states to adapt to the external environment. “In other words, the general theory of social systems claims to encompass all sociology’s potential topics and, in this sense, to be a universal sociological theory.” Universal sociological theories are terribly unpopular these days. But Luhmann attempted it. Did he succeed?

“Yet, a claim to universality is not a claim to exclusive correctness, to the exclusive validity, and thus necessity (noncontingency), of one’s own account.” Nobody claiming to have a universal theory does this. Indeed, a theory learns about its own contingency through self-reference. So, social systems theory discovers it European origins, for example, as soon as it considers itself. What then? At that point, one “distinguish[es] between claims of universality and claims to exclusivity”, which makes utter sense, or “by recognizing that structural contingencies must be employed as an operative necessity, with the consequence that there is a constant contingency absorbtion through the successes, practices, and commitments in the scientific system.”

Contingency absorbtion is a nice idea. It is perhaps associated with the idea of abstraction: as one accumulates contingent experiences and abstracts from them, one discovers necessary generalities which are true for all contingent experiences. This has been the core German philosophical method for centuries, and it is quite powerful. We seem to have completely forgotten it in the American academic system. That is why the computer scientists have taken over everything. They have a better universalizing science than the sociologists do. Precisely for that reason, we are seeing computational systems in constant and irksome friction with society. American sociologists need to stop insisting on theoretical pluralism and start developing a universal sociology that is competitive, in terms of its universality, with computer science, or else we will never get smart infrastructure and AI ethics right.

References

Luhmann, N. (1995). Social systems. Stanford University Press.

Leave a comment

February 21, 2021

Social justice, rationalism, AI ethics, and libertarianism

There has been a lot of drama on the Internet about AI ethics, and that drama is in the mainstream news. A lot of this drama is about the shakeup in Google’s AI Ethics team. I’ve already written a little about this. I’ve been following the story since, and I would adjust my emphasis slightly. I think the situation is sadder than I did at first. But my take (and I’m sticking to it) is that what’s missing from the dominant narrative in that story, that it is about race and gender representation in tech, (a view quite well articulated in academic argot by Tao and Varshney (2021), by the way) is an analysis of the corporate firm as an organization, what it means to be an employee of a firm, and the relationship between firms and artificial intelligence.

These questions about the corporate firm are not topics that get a lot of traction on the Internet. The topics that get a lot of traction on the Internet are race and gender. I gather that this is something anybody with a blog, (let alone publishing news), discovers. I write this blog for an imagined audience of about ten people. It is mainly research notes to myself. But for some reason, this post about racism got 125 views this past week. For me, that’s a lot of views. Who are those people?

The other dramatic thing going in my corner of the Internet this past week is also about race and gender, and also maybe AI ethics. It is the reaction to Cade Metz’s NYT article “Silicon Valley’s Safe Space“, which is about the blog Slate Star Codex (SSC), rationalist subculture, libertarianism and … racism and sexism.

I learned about this kerfuffle in a backwards way. I learned about it through Glen Weyl’s engagement with SSC about technocracy, which I suppose he was bumping to ride on the shockwave created by the NYT article. In 2019, Weyl posted a critique of “technocracy” which was also a rather pointed attack on the rationalism community, in part because of its connection to “neoreaction”. SSC responded rather adroitly, in my opinion; Weyl responded. It’s an interesting exchange about political theory.

I follow Weyl on Twitter because I disagree with him rather strongly on niche topics in data regulation. As a researcher, I’m all about these niche topics in data regulation; I think they are where the rubber really hits the road on AI ethics. Weyl has published the view that consumer internet data should be treated as a form of labor. In my corner of technology policy research, which is quite small in terms of its Internet footprint, we think this is nonsense; it is what Salome Viljoen calls a propertarian view of data. The propertarian view of data is, for many important reasons, wrong. “Markets are never going to fix the AI/data regulation problem. Go read some Katharina Pistor, for christ’s sake!” That’s the main thing I’ve been waiting to say to Weyl, who has a coveted Microsoft Research position, a successful marketing strategy for his intellectual brand, perfect institutional pedigree, and so on.

Which I mean to his credit: he’s a successful public intellectual. This is why I was surprised to see him tweeting about rationalist subculture, which is not, to me, a legitimate intellectual topic, sanctioned as part of the AI/tech policy/whatever research track. My experiences with rationalism have all been quite personal and para-academic. It is therefore something of a context collapse — the social media induced bridging of social spheres — for me personally. (Marwick and Boyd, 2011; Davis and Jurgenson, 2014)

Context collapse is what it’s all about, isn’t it? One of the controversies around the NYT piece about SSC was that the NYT reporter, Metz, was going to reveal the pseudonymous author of SSC, “Scott Alexander”, as the real person Scott Siskind. This was a “doxxing”, though the connection was easy to make for anybody looking for it. Nevertheless, Siskind initially had a strong personal sense of his own anonymity as a writer. Siskind is, professionally, a psychotherapist, which is a profession with very strong norms around confidentiality and its therapeutic importance. Metz’s article is about this. It is also about race and gender — in particular the ways in which there is a therapeutic need for a space in which to discuss race and gender, as well as other topics, seriously, but also freely, without the major social and professional consequences that have come to be associated with it.

Smart people are writing about this and despite being a “privacy scholar”, I’m not sure I can say much that is smarter. Will Wilkinson’s piece is especially illuminating. His is a defense of journalism and a condemnation of what, I’ve now read, was an angry Internet mob that went after Metz in response to this SSC doxxing. It is also an unpacking of Siskind’s motivations based on his writing, a diagnosis of sorts. Spiers has a related analysis. A theme of these arguments against SSC and the rationalists is, “You weren’t acting so rationally this time, were you?”

I get it. The rationalists are, by presenting themselves at times as smarter-than-thou, asking for this treatment. Certainly in elite intellectual circles, the idea that participation in a web forum should give you advanced powers of reason that are as close as we will ever get to magic is pretty laughable. What I think I can say from personal experience, though, is that elite intellectuals seriously misunderstand popular rationalism if they think that it’s about them. Rather, popular rationalism is a non-elite social movement that’s often for people with non-elite backgrounds and problems. Their “smarter-than-thou” was originally really directed at other non-elite cultural forms, such as Christianity (!). I think this is widely missed.

I say this with some anecdata. I lived in Berkeley for a while in graduate school and was close with some people who were bona fide rationalists. I had an undergraduate background from an Ivy League university in cognitive science and was familiar with the heuristics and biases program and Bayesian reasoning. I had a professional background working in software. I should have fit in, right? So I volunteered twice at the local workshop put on by the Center for Applied Rationality to see what it was all about.

I noticed, as one does, that it’s mostly white men at these workshops and when asked for feedback I pointed this out. Eager to make use of the sociological skills I was learning in grad school, I pointed out that if they did not bring in more diversity early on, then because of homophily effects they may never reach a diverse audience.

At the time, the leadership of CFAR told me something quite interesting. It was that they had looked at their organizational goals and capacities and decided that where they could make the most impact was on teaching the skills of rational thought to smart people from, say, the rural midwest U.S.A. who would otherwise not get the exposure to this kind of thinking or what I would call a community of practice around it. Many of these people (much like Elizabeth Spiers, according to her piece) come from conservative and cloistered Christian backgrounds. Yudkowsky’s Harry Potter and the Methods of Rationality is their first exposure to Bayesian reasoning. They are often the best math students in their homogeneous home town, and finding their way into an engineering job in California is a big deal, as is finding a community that fills an analogous role to organized religion but does not seem so intellectually backwards. I don’t think it’s accidental the Julia Galef, who founded CFAR, started out in intellectual atheist circles before becoming a leader in rationalism. Providing an alternative culture to Christianity is largely what popular rationalism is about.

From this perspective, it makes more sense why Siskind has been able to cultivate a following by discussing cultural issues from a centrist and “altruistic” perspective. There’s a population in the U.S. that grew up in conservative Christian settings, now makes a living in a booming technology sector whose intellectual principles are at odds with those of their upbringing, is trying to “do the right thing” and, being detached from political institutions or power, turns to the question of philanthropy, codified into Effective Altruism. This population is largely comprised of white guys who may truly be upwardly mobile because they are, relative to where they came from, good at math. The world they live in, which revolves around AI, is nothing like the one they grew up in. These same people are regularly confronted by a different ideology, a form of left wing progressivism, which denies their merit, resents their success, and considers them a problem, responsible for the very AI harms that they themselves are committed to solving. If I were one of them, I, too, would want to be part of a therapeutic community where I could speak freely about what was going on.

This is several degrees removed from libertarian politics, which I now see as the line connecting in Weyl. Wilkinson makes the compelling case for contemporary rationalism originating in Tyler Cohen’s libertarian economist blogging and the intellectual environment at George Mason University. This spins out Robin Hanson’s Overcoming Bias blog, which spins out Yudkowsky’s LessWrong forum, which is where popular rationalism incubated. Weyl is an east coast libertarian public intellectual and it makes sense that he would engage other libertarian public intellectuals. I don’t think he’s going to get very far picking fights on the Internet with Yudkowsky, but I could be wrong.

Weyl’s engagement with the rationalist community does highlight for me two other missing elements in the story-as-told-so-far, from my readings on it. I’ve been telling a story partly about geography and migration. I think there’s also an element of shifting centers of cultural dominance. Nothing made me realize that I am a parochial New Yorker like living in California for five years. Rationalism remains weird to me because it is, today, a connection between Oxford utilitarian philosophers, the Silicon Valley nouveau riche, and to some extent Washington, D.C.-based libertarians. That is a wave of culture bypassing the historical intellectual centers of northeast U.S. Ivy League universities which for much of America’s history dominated U.S. politics.

To some extent, this speaks to the significance of the NYT story as well. It was not the first popular article about rationalists; Metz mentions the TechCrunch article about neoreactionaries (I’ll get to that) but not the Sam Frank article in Harpers, “Come With Us If You Want To Live” (2015), which is more ethnographic in its approach. I think it’s a better article. But the NYT has a different audience and a different standard for relevance. NYT is not an intellectual literary magazine. It is the voice of New York City, once the Center of the Universe. New York City’s perspective is particularly weighty, relevant, objective, powerful because of its historic role as a global financial center and marketing center. When the NYT notices something, for a great many people, it becomes real. NYT is at the center of a large public sphere with a specific geographic locus, in a way that some blogs and web forums are not. So whether it was justified or not, Metz’s doxing of Siskind was a significant shift in what information was public, and to whom. Part of its significance is that it was an assertion of cultural power by an institution tied to old money in New York City over a beloved institution of new money in Silicon Valley. In Bourdieusian terms, the article shifted around social and cultural capital in a big way. Siskind was forced to make a trade by an institution more powerful than him. There is a violence to that.

This force of institutional power is perhaps the other missing element in this story. Wilkinson and Frank’s pieces remind me: this is about libertarianism. Weyl’s piece against technocracy is also about libertarianism, or maybe just liberalism. Weyl is arguing that rationalists, as he understands them, are libertarians but not liberals. A “technocrat” is somebody who wants to replace democratic governance mechanisms, which depend on pluralistic discourse, with an expert-designed mechanism. Isn’t this what Silicon Valley does? Build Facebook and act like it’s a nation? Weyl, in my reading, wants an engaged pluralistic public sphere. He is, he reveals later, really arguing with himself, reforming his own views. He was an economist, coming up with mathematical mechanisms to improve social systems through “radical exchange”; now he is a public intellectual who has taken a cultural turn and called AI an “ideology”.

On the other end of the spectrum, there are people that actually would, if they could, build an artificial island and rule it via computers like little lords. I guess Peter Thiel, who plays a somewhat arch-villain role in this story, is like this. Thiel does not like elite higher education and the way it reproduces the ideological conditions for a pluralistic democracy. This is presumably why he backs Curtis Yarvin, the “neoreactionary” writer and “Dark Enlightenment” thinker. Metz goes into detail about this, and traces a connection between Yarvin and SSC; there’s leaked emails about it. To some people, this is the real story. Why? Because neoreaction is racist and sexist. This, not political theory, I promise you, is what is driving the traffic. It’s amazing Metz didn’t use the phrase “red pill” or “alt-right” because that’s definitely the narrative being extended here. With Trump out of office and Amazon shutting down Parler’s cloud computing, we don’t need to worry about the QAnon nutcases (which were, if I’m following correctly, a creation of the Mercers) but what about the right wing elements in the globally powerful tech sector, because… AI ethics! There’s no escape.

Slate Star Codex was a window into the Silicon Valley psyche. There are good reasons to try and understand that psyche, because the decisions made by tech companies and the people who run them eventually affect millions.
And Silicon Valley, a community of iconoclasts, is struggling to decide what’s off limits for all of us.
At Twitter and Facebook, leaders were reluctant to remove words from their platforms — even when those words were untrue or could lead to violence. At some A.I. labs, they release products — including facial recognition systems, digital assistants and chatbots — even while knowing they can be biased against women and people of color, and sometimes spew hateful speech.
Why hold anything back? That was often the answer a Rationalist would arrive at.

Metz’s article has come under a lot of criticism for drawing sweeping thematic links between SSC, neoreaction, and Silicon Valley with very little evidence. Noah Smith’s analysis shows how weak this connection actually is. Silicon Valley is, by the numbers, mostly left-wing, and mostly, by the numbers, not reading rationalist blogs. Thiel, and maybe Musk, are noteworthy exceptions, not the general trend. What does any of this have to do with, say, Zuckerburg? Not much.

The trouble is that if the people in Silicon Valley are left-wing, then there’s nobody to blame for racist and sexist AI. Where could racism and sexism in AI possibly come from, if not some collective “psyche” of the technologists? Better, more progressive leaders in Silicon Valley, the logic goes, would lead to better social outcomes. Pluralistic liberalism and proper demographic representation would, if not for the likes of bad apples like Thiel, steer the AI Labs and the big tech companies that use their products towards equitability and justice.

I want to be clear: I think that affirmative action for under-represented minorities (URMs) in the tech sector is a wonderful thing, and that improving corporate practices around their mentorship, etc. is a cause worth fighting for. I’m not knocking any of that. But I think the idea that this alone will solve the problems of “AI ethics” is a liberal or libertarian fantasy. This is because assuming that the actions of a corporation will have the politics of its employees is a form of ecological fallacy. Corporations do not work for their employees; they work, legally and out of fiduciary duty, to their shareholders. And the AI operated at the grand social scales that we are talking about are not controlled by any one person; they are created and operated corporately.

In my view, what Weyl (who I probably agree with more than I don’t), the earlier libertarian bloggers like Hanson, the AI X-risk folks like Yudkowsky, and the popular rationalist movement all get wrong is the way institutional power necessarily exceeds that of individuals in part because and through of “artificial intelligence”, but also through older institutions that distribute economic and social capital. The “public sphere” is not a flat or radical “marketplace of ideas”; it is an ecology of institutions like the New York Times, playing on ideological receptiveness grounded in religious and economic habitus.

Jeffrey Friedman is a dedicated intellectual and educator, who for years has been a generous intellectual mentor and facilitator of political thought. Friedman, like Weyl, is a critic of technocracy. In an early intellectual encounter that was very formative for me, he invited me to write a book review of Philip Tetlock Expert Political Judgment for his journal. In 2007, it was my first academic publication. The writing is embarrassing and I’m glad it is behind a paywall. In the article, I argue against Friedman and for technocracy based on the use of artificial intelligence. I have been in friendly disagreement on this point with Friedman ever since.

The line of reasoning from Friedman’s book, Power without Knowledge: A Critique of Technocracy (2019), that I find most interesting is about the predictability of individuals. Is it possible for a technocrat to predict society? This question has been posed by many different social theorists. I believe Friedman is unique in suggesting that (a) individuals cannot be predicted by a technocrat because (b) of how individual behavior is determined by their ideas, which are sourced so variously and combined in such complex ways that they cannot be captured by an external or generalizing observer. The unpredictability of society is an objective obstacle to the social scientific prediction that is required for technocratic policies to function effectively and as intended. Instead of a technocracy, Friedman advocates for an “Exitocracy”, based on the idea of Exit from Hirschman, which prioritizes a robust private sector in which citizens can experiment and find happiness, over technocratic or what others might call paternalist public policy. Some of the attractiveness of this model is that it depends on minimal assumptions about the rationality of agents, and especially the agency of technocrats, but still achieves satisficing results. Friedman’s exitocracy is, he argues, a ‘judicious’ technocracy, calibrated to realistic levels of expertise and public ignorance. In allowing for redistribution and centralized governance of some public goods, Friedman’s exitocracy stands as an alternative to more radically libertarian “Exit”-oriented proposals such as those of Srinivasan, which have been associated with Silicon Valley and, by dubious extension, the rationalist community.

At this time, I continue to disagree with Friedman. Academics are particularly unpredictable with respect to their ideas, especially if they do not value prestige or fame as much as their other colleagues. But most people are products of their background, or their habits, or their employers, or their social circles, or their socially structured lived experience. Institutions can predict and control people, largely by offering economic incentives. This predictability is what has made AI effective commercially — in uses by advertisers, for example — and it was what makes centralize public policy, or technocracy, possible.

But I could be wrong about this point. It is an empirical question how and under what conditions people’s beliefs and actions are unpredictable given background conditions. This amount of variability, which we might consider a kind of “freedom”, is a condition for whether technocracy, and AI, is viable or not. Are we free?

References

Davis, J. L., & Jurgenson, N. (2014). Context collapse: Theorizing context collusions and collisions. Information, communication & society, 17(4), 476-485.

Friedman, J. (2019). Power without knowledge: a critique of technocracy. Oxford University Press.

Kollman, K., Miller, J. H., & Page, S. E. (1997). Political institutions and sorting in a Tiebout model. The American Economic Review, 977-992.

Marwick, A. E., & Boyd, D. (2011). I tweet honestly, I tweet passionately: Twitter users, context collapse, and the imagined audience. New media & society, 13(1), 114-133.

Pistor, K. (2020). Rule by data: The end of markets?. Law & Contemp. Probs., 83, 101.

Tao, Y., & Varshney, K. R. (2021). Insiders and Outsiders in Research on Machine Learning and Society. arXiv preprint arXiv:2102.02279.

4 Comments

February 16, 2021

Crypto, macro, and information law

Dogecoin is in the news this week because of Elon Musk’s pump and dump in the latest of notable asset bubbles fueled in small part by Internet-informed, perhaps frivolous, day-traders. The phenomenon reminds me of this curious essay about viral art. It concludes:

The doge meme is a Goldsmithian piece, passing ephemerally through a network of peers. In a LaBeoufian moment, Jackson Palmer invented Dogecoin, capturing the meme and using it to leverage networks of power. Now it is art post-LaBeouf in its greatest form: authorless art as economic power, transmitted over networks. As the synthesized culmination of the traditions of economics and Western literature, DogeCoin is one of the greatest achievements in the history of art, if not the greatest.

This paragraph is perhaps best understood, if at all, as an abstruse joke. The essay itself is most likely not written by “Niklos Szabo”, easily conflated with Nick Szabo, one of the deeper thinkers behind cryptocurrency more generally. The real Szabo has written much more seriously and presciently about culture and the economy. As an aside, I believe Szabo’s writings about book consciousness prefigure Hildebrandt’s work on the role of the printing press as a medium contributing to the particular character of text-driven law. However if the enduring success of cryptocurrencies validates Szabo’s economics more than his cultural theory. His 2002 paper “Shelling out: the origins of money” is a compelling history of currency. Notably, it is not a work of formal economic theory. Rather, it draws on historical and anthropological examples to get at the fundamentals of the role currency plays in society. This study leads to the conclusion that currency must be costly to created and transferable with relatively low transaction costs. Bitcoin, for example, was designed to have these qualities.

What Szabo does not discuss in “Shelling out” is the other thing Bitcoin is most known for, which is speculative asset bubble pricing. Cryptocurrency has lurched into the mainstream in fits of speculative enthusiasm followed by crashes and breakdowns. It is risky.

Salome Viljoen and I are writing about financial regulations as part of our “Data Market Discipline” project. One takeaway from this work is that the major financial regulations in the United States were responses to devastating financial crises, such as the Great Depression and the 2008 financial crisis, which were triggered by the collapse of an asset bubble. So while currency is an old invention and the invention of new currencies is interesting, the project of maintaining a stable financial system is a relatively more recent legal project and an unfinished one at that. It is so much more unfinished for cryptocurrencies, that are not controlled by a central banking system, than for national fiat currencies for which, for example, interest rates can be used as a calibrating tool.

These are not idle theoretical points. Rather, they are at the heart of questions surrounding the recovery of the economy from COVID-related setbacks. Money from stimulus checks going to people who have no reason to increase their consumption (cf. Carroll et al., 2020) is perhaps responsible for the influx of retail investment into equities markets and, in particular, Reddit-coordinated asset bubbles such as the ones we’re seeing recently with Gamestop and Dogecoin. The next stimulus package being prepared by the Biden administration has sounded alarms from parts of the Economics establishment that it will spur inflation, while Janet Yellen has argued that this outcome can be prevented using standard monetary policy tools such as the increase of interest rates. Arguably, the recent rising price of Bitcoin is due to this threat of macro-economic stability of the dollar-denominated financial system.

I don’t mean any of this conclusively. Rather, I’m writing this to register my growing realization that the myriad Internet effects on culture, economy, and the law are often much more motivated by movements in internationally coupled financial systems than “technology policy” specialists or “public interest technologists” are inclined to admit. We are inclined, because of our training in something else — whether it be computer science, environmental law, political philosophy, or whatever — to seek out metaphors from our own domain of expertise. But many of the most trenchant analyses of why the current technological landscape seems a bit off come down to failures of the price mechanism in the digital economy. I’m thinking of Kapczynski ‘s (2011) critique of the price mechanism in relation to intellectual property, and Strandburg’s (2013) analysis of the failure of pricing in on-line services. We have on the one hand the increasingly misconceptualized “Silicon Valley”‘s commitment to a “free market” and on the other hand few of the conditions under which a “free market” is classically considered to be efficient. The data economy does not meet even classically liberal (let alone New, more egalitarian, Liberal) standards of justice. And liberal legal theory is not equipped, Jake Goldenfein and I have argued, to grapple with this reality.

What progress can be made?

Maybe there is something somebody with enormous wealth or institutional power could do to change the situation. I’m not one of those people. However, there is some evidence to support the point that at the root of these problems is a conceptual, intellectual failing to understand what’s going on at the root of it.

In some recent work with Kathy Strandburg, we are examining the conceptual roots of the highly influential Law and Economics (L&E) branch of legal scholarship. This field absorbs the techniques of neoclassical economics and develops them into actionable policy proposals and legal rules of thumb. It has come under political criticism from the recently formed Law and Political Economy (LPE) movement. Interestingly, it has also been critique from a “Law and Macroeconomics” perspective, which argues that L&E should really be called “law and microeconomics”, because of its inability to internalize macroeconomic concepts such as the business cycle or changes in monetary policy.

Among the assumptions at the roots of L&E are notions of optimality and efficiency that make somewhat naive assumptions about the nature of price and money. For example, Kaldor-Hicks efficiency, a relaxation of Pareto efficiency used in welfare economics as applied to L&E, allows for transactions that alter the situations of agents so long as one agent, who gains, could theoretically compensate the other for their losses (see Feldman, 1998), . This concept is used to consider social welfare optimal, resolving the neoclassical problem of the incomparability of individual utilities through an implicit pricing mechanism. This leads L&E to favor “wealth maximizing” policies.

However, grounding legal theory in the idea of a robust price mechanism capable of subsuming all differences in individual preferences is quite naive in a digital economy that is always already at the intersection of many different currencies (including cryptocurrency), variable and politically vulnerable systems of credit and debt, and characterized by markets that do not that the legal scaffolding needed to drive them towards “true” prices. If Mirowski and Nik-Khah (2017) are correct and Economists have abandoned earlier notions of “truth” to faith in the market’s price as a “truth” derived from streams of information, something is indeed amiss. Data is not a commodity, and regulations that treat data flows as commodity exchanges not well matched to the reality. In the Hayekian model, price is the signal that combines available information. In the data economy, the complexity topology of real data flows belies simplistic views of “the market”.

What tech law needs is a new economic model, one that, just as general relativity in physics showed how classical mechanics was a special case of more complex universal laws, reveals how data, intellectual property, and price are connected in ways that go beyond the classical liberal imagination.

References

Benthall, Sebastian and Viljoen, Salome, Data Market Discipline: From Financial Regulation to Data Governance (January 27, 2021). J. Int’l & Comparative Law – (2021)

Carroll, C. D., Crawley, E., Slacalek, J., & White, M. N. (2020). Modeling the consumption response to the CARES Act (No. w27876). National Bureau of Economic Research.

Feldman, A. M. (1998). Kaldor-hicks compensation. The new Palgrave Dictionary of economics and the law, 2, 417-421.

Hildebrandt, M. (2015). Smart technologies and the end (s) of law: novel entanglements of law and technology. Edward Elgar Publishing.

Kapczynski, A. (2011). The cost of price: Why and how to get beyond intellectual property internalism. UCLA L. Rev., 59, 970.

Mirowski, P., & Nik-Khah, E. (2017). The knowledge we have lost in information: the history of information in modern economics. Oxford University Press.

Shekman, David. “Gamestop and the Surrounding Legal Questions.” Medium, Medium, 5 Feb. 2021, medium.com/@shekman27/gamestop-and-the-surrounding-legal-questions-fc0d1dc142d7.

Strandburg, K. J. (2013). Free fall: The online market’s consumer preference disconnect. U. Chi. Legal F., 95.

Szabo, N. (2002). Shelling out: the origins of money. Satoshi Nakamoto Institute.

Szabo, Niklos. “Art Post-LaBeouf.” Medium, Medium, 22 Sept. 2014, medium.com/@niklosszabo/art-post-labeouf-b7de5732020c.

Leave a comment

January 15, 2021

Hildebrandt (2013) on double contingency in Parsons and Luhmann

I’ve tried to piece together double contingency before, and am finding myself re-encountering these ideas in several projects. I just now happened on this very succinct account of double contingency in Hildebrandt (2013), which I wanted to reproduce here.

Parsons was less interested in personal identity than in the construction of social institutions as proxies for the coordination of human interaction. His point is that the uncertainty that is inherent in the double contingency requires the emergence of social structures that develop a certain autonomy and provide a more stable object for the coordination of human interaction. The circularity that comes with the double contingency is thus resolved in the consensus that is consolidated in sociological institutions that are typical for a particular culture. Consensus on the norms and values that regulate human interaction is Parsons’s solution to the problem of double contingency, and thus explains the existence of social institutions. As could be expected, Parsons’s focus on consensus and his urge to resolve the contingency have been criticized for its ‘past-oriented, objectivist and reified concept of culture’, and for its implicitly negative understanding of the double contingency.

This paragraph says a lot, both about “the problem” posed by “the double contingency”, the possibility of solution through consensus around norms and values, and the rejection of Parsons. It is striking that in the first pages of this article, Hildebrandt begins by challenging “contextual integrity” as a paradigm for privacy (a nod, if not a direct reference, to Nissenbaum (2009)), astutely pointing out that this paradigm makes privacy a matter of delinking data so that it is not reused across contexts. Nissenbaum’s contextual integrity theory depends rather critically on consensus around norms and values; the appropriateness of information norms is a feature of sociological institutions accountable ultimately to shared values. The aim of Parsons, and to some extent also Nissenbaum, is to remove the contingency by establishing reliable institutions.

The criticism of Parsons as being ‘past-oriented, objectivist and reified’ is striking. It opens the question whether Parsons’s concept of culture is too past-oriented, or if some cultures, more than others, may be more past-oriented, rigid, or reified. Consider a continuum of sociological institutions ranging from the rigid, formal, bureaucratized, and traditional to the flexible, casual, improvisational, and innovative. One extreme of these cultures is better conceptualized as “past-oriented” than the other. Furthermore, when cultural evolution becomes embedded in infrastructure, no doubt that culture is more “reified” not just conceptually, but actually, via its transformation into durable and material form. That Hildebrandt offers this criticism of Parsons perhaps foreshadows her later work about the problems of smart information communication infrastructure (Hildebrandt, 2015). Smart infrastructure poses, to those which this orientation, a problem in that it reduces double contingency by being, in fact, a reification of sociological institutions.

“Reification” is a pejorative word in sociology. It refers to a kind of ideological category error with unfortunate social consequences. The more positive view of this kind of durable, even material, culture would be found in Habermas, who would locate legitimacy precisely in the process of consensus. For Habermas, the ideals of legitimate consensus through discursively rational communicative actions finds its imperfect realization in the sociological institution of deliberative democratic law. This is the intellectual inheritor of Kant’s ideal of “perpetual peace”. It is, like the European Union, supposed to be a good thing.

So what about Brexit, so to speak?

Double contingency returns with a vengeance in Luhmann, who famously “debated” Habermas (a more true follower of Parsons), and probably won that debate. Hildebrandt (2013) discusses:

A more productive understanding of double contingency may come from Luhmann (1995), who takes a broader view of contingency; instead of merely defining it in terms of dependency he points to the different options open to subjects who can never be sure how their actions will be interpreted. The uncertainty presents not merely a problem but also a chance; not merely a constraint but also a measure of freedom. The freedom to act meaningfully is constraint [sic] by earlier interactions, because they indicate how one’s actions have been interpreted in the past and thus may be interpreted in the future. Earlier interactions weave into Luhmann’s (1995) emergent social systems, gaining a measure of autonomy — or resistance — with regard to individual participants. Ultimately, however, social systems are still rooted in double contingency of face-to-face communication. The constraints presented by earlier interactions and their uptake in a social system can be rejected and renegotiated in the process of anticipation. By figuring out how one’s actions are mapped by the other, or by social systems in which one participates, room is created to falsify expectations and to disrupt anticipations. This will not necessarily breed anomy, chaos or anarchy, but may instead provide spaces for contestation, self-definition in defiance of labels provided by the expectations of others, and the beginnings of novel or transformed social institutions. As such, the uncertainty inherent in the double contingency defines human autonomy and human identity as relational and even ephemeral, always requiring vigilance and creative invention in the face of unexpected or unreasonably constraining expectations.

Whereas Nissenbaum’s theory of privacy is “admitted conservative”, Hildebrandt’s is grounded in a defense of freedom, invention, and transformation. If either Nissenbaum or Hildebrandt were more inclined to contest each other directly, this may be privacy scholarship’s equivalent of the Habermas/Luhmann debate. However, this is unlikely to occur because the two scholars operate in different legal systems, reducing the stakes of the debate.

We must assume that Hildebrandt, in 2013, would have approved of Brexit, the ultimate defiance of labels and expectations against a Habermasian bureaucratic consensus. Perhaps she also, as would be consistent with this view, has misgivings about the extraterritorial enforcement of the GDPR. Or maybe she would prefer a a global bureaucratic consensus that agreed with Luhmann; but this is a contradiction. This psychologistic speculation is no doubt unproductive.

What is more productive is the pursuit of a synthesis between these poles. As a liberal society, we would like our allocation of autonomy; we often find ourselves in tension with the the bureaucratic systems that, according to rough consensus and running code, are designed to deliver to us our measure of autonomy. Those that overstep their allocation of autonomy, such as those that participated in the most recent Capitol insurrection, are put in prison. Freedom cooexists with law and even order in sometimes uncomfortable ways. There are contests; they are often ugly at the time however much they are glorified retrospectively by their winners as a form of past-oriented validation of the status quo.

References

Hildebrandt, M. (2013). Profile transparency by design?: Re-enabling double contingency. Privacy, due process and the computational turn: The philosophy of law meets the philosophy of technology, 221-46.

Hildebrandt, M. (2015). Smart technologies and the end (s) of law: novel entanglements of law and technology. Edward Elgar Publishing.

Nissenbaum, H. (2009). Privacy in context: Technology, policy, and the integrity of social life. Stanford University Press.

Leave a comment

January 14, 2021

Review: Software Development and Reality Construction

I’ve discovered a wonderful book, Floyd et al.’s “Software Development and Reality Construction” (1992). One of the authors has made a PDF available on-line. It represents a strand of innovative thought in system design that I believe has many of the benefits of what has become “critical HCI” in the U.S. without many of its pitfalls. Is is a playful compilation with many interesting intellectual roots.

From its blurb:

The present book is based on the conference Software Development and Reality Construction held at SchloB Eringerfeld in Germany, September 25 – 30, 1988. This was organized by the Technical University of Berlin (TUB) in cooperation with the German National Research Center for Computer Science (GMD), Sankt Augustin, and sponsored by the Volkswagen Foundation whose financial support we gratefully acknowledge. The conference was an interdisciplinary scientific and cultural event aimed at promoting discussion on the nature of computer science as a scientific discipline and on the theoretical foundations and systemic practice required for human-oriented system design. In keeping with the conversational style of the conference, the book comprises a series of individual contributions, arranged so as to form a coherent whole. Some authors reflect on their practice in computer science and system design. Others start from approaches developed in the humanities and the social sciences for understanding human learning and creativity, individual and cooperative work, and the interrelation between technology and organizations. Thus, each contribution makes its specific point and can be read on its own merit. But, at the same time, it takes its place as a chapter in the book, along with all the other contributions, to give what seemed to us a meaningful overall line of argumentation. This required careful editorial coordination, and we are grateful to all the authors for bearing with us throughout the slow genesis of the book and for complying with our requests for extensive revision of some of the manuscripts.

There are a few specific reasons I’m excited about this book.

First, it is explicitly about considering software development as a designing activity that is an aspect of computer science. In many U.S. scholarly contexts, there is an educational/research thrust towards removing “user interface design” from both the theoretical roots of computer science and the applied activity of software development. This has been a problem for recent scholarly debates about, for example, the ethics of data science and AI. When your only options are a humanities oriented “design” field, and a field of computer science “algorithms”, there is no room to explore the embodied practice of software development, which is where the rubber hits the road.

Second, this book has some fascinating authors. It includes essays from Heinz von Foerster, a second-order cybernetics Original Gangster. It also includes essays from Joseph Goguen, who is perhaps the true link between computer science theory (he was a theorist and programming language designer) and second-order cybernetics (Mutarana and Varela, which would then influence Winograd and Flores’s critique of AI, but also Niklas Luhmann, which shows up in other critiques of AI from a legal perspective). Indeed, Goguen co-authored papers with Varela (1979) formalizing Varela’s notions of autonomy and autopoiesis in terms of category theory — a foundation that has had little uptake since. But this is not a fringe production. Donald Knuth, a computer science god-king, has an essay in the book about the process of creating and debugging TeX, the typesetting language. It is perhaps not possible to get deeper into the heart of the embodied practice of technical work than that. His essay begins with a poem from Piet Hein:

The road to wisdom?
Well, it’s plain
and simple to express:
Err
and err
and err again
but less
and less
and less.

The book self-recognizes its interesting intellectual lineage. The following diagram is included in Raeithel’s article “Activity theory as a foundation for design”, which stakes out a Marxist Vygotskian take on design practice. This is positioned as an extreme view, to the (literally, on the page) left of the second-order cybernetics approach, which he positions as culminating in Winograd and Flores.

It is a sweeping, thoughtful book. Any one of its essays could, if more widely read, be a remedy for the kinds of conceptual errors made in today’s technical practice which lead to “unethical” or adverse outcomes. For example, Klein and Lyytinen’s “Towards a new understanding of data modelling” swiftly rejects notions of “raw data” and instead describes a goal oriented, hermeneutic data modeling practice. What if “big data” techniques had been built on this this understanding?

The book ultimately does not take itself too seriously. It has the whimsical character that the field of computer science could have in those early days, when it was open to conceptual freedom and exploration. The book concludes with a script for a fantastic play that captures the themes and ideas of the conference as a whole:

This goes on for six pages. By the end, Alice discovers that she is in a “cyberworld”:

Oh, what fun it is. It’s a huge game that’s being played – all over this
cyberworld – if this is a world at all. How I wish I was one of them! I
wouldn’t mind being a Hacker, if only I might join – though of course I
should like to be a Cyber Queen, best.

I’ve only scratched the surface of this book. But I expect to be returning to it often in future work.

References

Christiane Floyd, Heinz Züllighoven, and Reinhard Budde, Reinhard Keil-Slawik. (1992) “Software development and reality construction.” Springer-Verlag Berlin Heidelberg. https://doi.org/10.1007/978-3-642-76817-0

Goguen, J. A., & Varela, F. J. (1979). Systems and distinctions; duality and complement arity. International Journal of General System, 5(1), 31-43.

Klein, H. K., & Lyytinen, K. (1992). Towards a new understanding of data modelling. In Software development and reality construction (pp. 203-219). Springer, Berlin, Heidelberg.

Raeithel, A. (1992). Activity theory as a foundation for design. In Software development and reality construction (pp. 391-415). Springer, Berlin, Heidelberg.

2 Comments

January 1, 2021

Reflections on the Gebru/Google dismissal

I’ve decided to write up some reactions to the dismissal of Dr. Gebru from Google’s Ethical AI team. I have hesitated thus far because the issues revolve around a particular person who has a right to privacy, because of the possible professional consequences of speaking out (this research area is part of my professional field and the parties involved, including Google’s research team and Dr. Gebru, are all of greater stature in it than myself), because there is much I don’t know about the matter (I have no inside view of the situation at Google, for example), because the facts of the case look quite messy to me, with many different issues at stake, and because the ethical issues raised by the case are substantive and difficult. It has also been a time with pressing personal responsibilities and much needed holiday rest.

I’m also very aware that one framing of the event is that it is about diversity and representation within the AI ethics research community. There are some that believe that white etc. men are over-represented in the field. Implicitly, if I write publicly about this situation, representing, as it were, myself, I am part of that problem. More on that in a bit.

Despite all of these reasons, I think it is best to write something. The event has been covered by many mainstream news outlets, and Dr. Gebru has been about as public as is possible with her take on the situation. She is, I believe, a public figure in this respect. I’ve written before on related topics and controversies within this field and have sometimes been told by others that they have found my writing helpful. As for the personal consequences to myself, I try to hold myself to a high standard of courage in my research work and writing. I wouldn’t be part of this technology ethics field if I did not.

So what do I think?

First, I think there has been a lot of thoughtful coverage of the incident by others. Here are some links to that work. So far, Hanna and Whitaker‘s take is the most forceful in its analysis of the meaning of the incident for a “crisis in AI”. In their analysis:

There is a crisis, which involves:
- A mismatch between those benefiting from and creating AI — “the corporations and the primarily white male researchers and developers” — and those most likely to be harmed by AI — “BIPOC people, women, religious and gender minorities, and the poor” because of “structural barriers”. A more diverse research community is needed to “[center] the perspectives and experiences of those who bear the harms of these technologies.”
- The close ties between tech companies and ostensibly independent academic institutions that homogenize the research community, obscure incentives, and dull what might be a more critical research agenda.
To address this crisis:
- Tech workers should form an inclusive union that pushes back on Big Tech for ethical concerns.
- Funding for independent critical research, with greater guaranteed access to company resources, should be raised through a tax on Big Tech.
- Further regulations should be passed to protect whistleblowers, prevent discrimination, consumer privacy and the contestability of AI systems.

These lines of argument capture most of what I’ve seen more informally in Twitter conversations about this issue. As far as their practical recommendations go, I think a regulatory agency for Big Tech, analogous to the Securities Exchange Commission for the financial sector, with a federal research agency analogous to the Office of Financial Research, is the right way to go on this. I’m more skeptical about the idea of a tech workers union, but that this not the main focus of this post. This post is about Dr. Gebru’s dismissal and its implications.

I think it’s best if I respond to the situation we a series of questions.

First, was Dr. Gebru wrongfully terminated from Google? Wrongful termination is when an employer terminates a contract with an employee in retaliation for an anti-discrimination or whistleblowing action. The heart of the matter is that Dr. Gebru’s dismissal “smells like” wrongful termination: Dr. Gebru was challenging Google’s diversity programs internally; she was reporting environmental costs of AI in her research in a way that was perhaps like whistleblowing. The story is complicated by the fact that she was negotiating with Google, with the possibility of resignation as leverage, when she was terminated.

I’m not a lawyer. I have come to appreciate the importance of the legal system rather late in my research career. Part of that appreciation is of how the law has largely anticipated the ethical issues raised by “AI” already. I am surprised, however, that the phrase “wrongful termination” has not been raised in journalism covering Dr. Gebru’s dismissal. It seems like the closest legal analog. Could, say, a progressively orientated academic legal clinic help Dr. Gebru sue Google over this? Does she have a case?

These are not idle questions. If the case is to inform better legal protection of corporate AI researchers and other ‘tech workers’, then it is important to understand the limits of current wrongful termination law, whether these limits cover the case of Dr. Gebru’s dismissal, and if not, what expansions to this law would be necessary to cover it.

Second, what is corporate research (and corporate funded research) for? The field of “Ethical AI” has attracted people with moral courage and conviction who probably could be doing other things if they did not care so much. Many people enter academic research hoping that they can somehow, through their work, make the world a better place. The ideal of academic freedom is that it allows researchers to be true to their intellectual commitments, including their ethical commitments. It is probably true that “critical” scholarship survives better in the academic environment. But what is corporate research for? Should we reasonably expect a corporation’s research arm to challenge that corporation’s own agendas?

I’ve done corporate research. My priorities were pretty clear in that context: I was supposed to make the company I worked for look smart. I was supposed to develop new technical prototypes that could be rolled into products. I was supposed to do hard data wrangling and analysis work to suss out what kinds of features would be possible to build. My research could make the world a better place, but my responsibility to my employer was to make it a better place by improving our company’s offerings.

I’ve also done critical work. Critical work tends not to pay as well as corporate research, for obvious reasons. I’m mainly done this from academic positions, or as a concerned citizen writing on my own time. It is striking that Hanna and Whitaker’s analysis follows through to the conclusion that critical researchers want to get paid. Their rationale is that society should reinvest the profits of Big Tech companies into independent research that focuses on reducing Big Tech harms. This would be like levying a tax on Big Tobacco to fund independent research into the health effects of smoking. This really does sound like a good idea to me.

But this idea would sound good to me even without Dr. Gebru’s dismissal from Google. To conflate the two issues muddies the water for me. There is one other salient detail: some of the work that brought Dr. Gebru into research stardom was now well-known audits of facial recognition technology developed by IBM and Microsoft. Google happily hired her. I wonder if Google would have minded if Dr. Gebru continued to do critical audits of Microsoft and IBM from her Google position. I expect Google would have been totally fine with this: one purpose of corporate research could be digging up dirt on your competition! This implies that it’s not entirely true that you can’t do good critical work from a corporate job. Maybe this kind of opposition research should be encouraged and protected (by making Big Tech collusion to prevent such research illegal).

Third, what is the methodology of AI ethics research? There are two schools of thought in research. There’s the school of thought that what’s most important about research is the concrete research question and that any method that answers the research question will do. Then there’s the school of thought that says what’s most important about research is the integrity of research methods and institutions. I’m of the latter school of thought, myself.

One thing that is notable about top-tier AI ethics research today is the enormously broad interdisciplinary range of its publication venues. I would argue that this interdisciplinarity is not intellectually coherent but rather reflects the broad range of disciplinary and political interests that have been able to rally around the wholly ambiguous idea of “AI ethics”. It doesn’t help that key terms within the field, such as “AI” and “algorithm”, are distorted to fit whatever agenda researchers want for them. The result is a discursive soup which lacks organizing logic.

In such a confused field, it’s not clear what conditions research needs to meet in order to be “good”. In practice, this means that the main quality control and/or gatekeeping mechanism, the publishing conferences, operate through an almost anarchic process of peer review. Adjacent to this review process is the “disciplinary collapse” of social media, op-eds, and whitepapers, which serve various purposes of self-promotion, activism/advocacy, and marketing. There is little in this process to incentivize the publication of work that is correct, or to set the standards of what that would be.

This puts AI ethics researchers in a confusing position. Google, for example, can plausible set its own internal standards for research quality because the publication venues have not firmly set their own. Was Dr. Gebru’s controversial paper up to Google’s own internal publication standards, as Google has alleged? Or did they not want their name on it only because it made them look bad? I honestly don’t know. But even though I have written quite critically about corporate AI “ethics” approaches before, I actually would not be surprised if a primarily “critical” researcher did not do a solid literature review of the engineering literature on AI energy costs before writing a piece about it, because the epistemic standards of critical scholarship and engineering are quite different.

There has been a standard floated implicitly or explicitly by some researchers in the AI ethics space. I see Hanna and Whitaker as aligned with this standard and will borrow their articulation. In this view, the purpose of AI ethics research is to surface the harms of AI so that they may be addressed. The reason why these harms are not obvious to AI practitioners already is the lack of independent critical scholarship by women, BIPOC, the poor, and other minorities. Good AI ethics work is therefore work done by these minorities such that it expresses their perspective, critically revealing faults in AI systems.

Personally, I have a lot of trouble with this epistemic standard. According to it, I really should not be trying to work on AI ethics research. I am simply, by fault of my subject position, unable to do good work. Dr. Gebru, a Black woman, on the other hand, will always do good work according to this standard.

I want to be clear that I have some of Dr. Gebru’s work and believe it deserves all of its accolades for reasons that are not conditional on her being a Black woman. I also understand why her subject position has primed her to do the kind of work that she has done; she is a trailblazer because of who she is. But if the problem faced by the AI ethics community is that its institutions have blended corporate and academic research interests so much that the incentives are obscure and the playing field benefits the corporations, who have access to greater resources and so on, then this problem will not be solved by allowing corporations to publish whatever they want as long as the authors are minorities. This would be falling into the trap of what Nancy Fraser calls progressive neoliberalism, which incentivizes corporate tokenization of minorities. (I’ve written about this before.)

Rather, the way to level the playing field between corporate research and independent or academic research is to raise the epistemic standard of the publication venues in a way that supports independent or academic research. Hanna and Whitaker argue that “[r]esearchers outside of corporate environments must be guaranteed greater access to technologies currently hidden behind claims of corporate secrecy, such as access to training data sets, and policies and procedures related to data annotation and content moderation.” Nobody, realistically, is going to guarantee outside researchers access to corporate secrets. However, research publication venues (like conferences) can change their standards to mandate open science practices: access to training data sets, reproducibility of results, no dependence on corporate secrets, and so on.

A tougher question for AI ethics research in particular is the question of how to raise epistemic standards for normative research in a way that doesn’t beg the question on interpretations of social justice or devolve into agonistic fracturing on demographic grounds. There are of course academic disciplines with robust methods for normative work; they are not always in communication with each other. I don’t think there’s going to be much progress in the AI ethics field until a sufficient synthesis of feminist epistemology and STEM methods has been worked out. I fear that is not going to happen quickly because it would require dropping some of what’s dear to situated epistemologies of the progressive AI ethics wing. But I may be wrong there. (There was some work along these lines by methodologists some years ago under the label “Human-Centered Data Science”.)

Lastly, whatever happened to the problem of energy costs of AI, and climate change? To me, what was perhaps most striking about the controversial paper at the heart of Dr. Gebru’s dismissal was that it wasn’t primarily about representation of minorities. Rather, it was (I’ve heard–I haven’t read the paper yet) about energy costs of AI, which is something that, yes, even white men can be concerned about. If I were to give my own very ungenerous, presumptuous, and truly uninformed interpretation of what the goings-on at Google were all about, I would put it this way: Google hired Dr. Gebru to do progressive hit pieces on competitor’s AI products like she had done for Microsoft and IBM, and to keep the AI ethics conversation firmly in the territory of AI biases. Google has the resources to adjust its models to reduce these harms, get ahead of AI fairness regulation, and compete on wokeness to the woke market segments. But Dr. Gebru’s most recent paper reframes the AI ethics debate in terms of a universal problem of climate change which has a much broader constituency, and which is actually much closer to Google’s bottom line. Dr. Gebru has the star power to make this story go mainstream, but Google wants to carve out its own narrative here.

It will be too bad if the fallout of Dr. Gebru’s dismissal is a reversion of the AI ethics conversation back to the well-trod questions of researcher diversity, worker protection, and privacy regulation, when the energy cost and climate change questions provide a much broader base of interest from which to refine and consolidate the AI ethics community. Maybe we should be asking: what standards should conferences be holding researchers to when they make claims about AI energy costs? What are the standards of normative argumentation for questions of carbon emission, which necessarily transcend individual perspectives, while of course also impacting different populations disparately? These are questions everybody should care about.

EDIT: I’m sensitive to the point that suggesting that Big Tech’s shift of the frame of AI ethics towards ‘fairness’ in a socially progressive sense is somewhat disingenuous may be seen as a rejection of those progressive politics, especially in the absence of evidence. I don’t reject those politics. This journalistic article by Karen Hao provides some evidence as to how another Big Tech company, Facebook, has deliberately kept AI fairness in the ethical frame and discouraged frames more costly to its bottom line, like the ethics of preventing disinformation.

Leave a comment