I’m pleased to say that now there is a 0.1.0 release of scikit-agent available.
Like one does, I find that as I work on it, what was a simple idea expands, and I now understand that the project is indeed quite technically ambitious. This first release is just a tiny part of what the library should be. Nevertheless, the announcement has had a positive reception among friends and even LinkedIn of all fora. It is an encouraging start.
It is very satisfying for me to build this tool, as it feels like the culmination of a lot of prior work. I was reminded this morning that as early as 2013 I was working on strategic graphical models — I hadn’t yet heard of Koller and Milch’s MAIDs or ‘structural causal games’ — out of a concern for “computational asymmetry” — market failures due to an imbalance in available compute infrastructure. I think I may finally be building the research tooling needed to do replicable work on that problem. This is just in time for a global arms race in data center build-out.
The same tooling is, I believe, the right path forward for establishing privacy and loyalty in Agentic AI, and in establishing safety in multi-agent AI systems.
I have one fear, which is that this is a hammer, and everything looks like a nail to me. However, I like to think that I have encountered a lot of strange nails and didn’t have the right tool for them, and that this is what’s informing the hammer design.
I will admit that AI code generation and agents are helping me a lot with the design and implementation of the software. This was a source of discomfort early on. My conclusion is that the key to using these tools is responsible workflow.
I’m very grateful to Alan Lujan for his contributions to the library. scikit-agent is quite literally a fork of Econ-ARK’s HARK library, which is where Alan and I started working together. There are a lot of techniques and use cases for scikit-agent that are based on heterogeneous agent macroeconomic modeling, which is a very technically deep field with its own depths of specialization. Alan and the computational economists are able to see a lot of things that I, with my rather interdisciplinary and scattered training, have as blind spots. I’m always learning in my interactions with him.
I’m likewise very grateful to the now rather dormant Causal Incentives Working Group, which between 2019 and 2025 did a lot of foundational work on causal game theory, which really put it back on the map of AI research after a long hiatus 2005-2015. I only wish I had been able to connect with them earlier and be involved with some of their great work. Now it’s a pleasure to build on it. I expect scikit-agent to eventually include a lot of examples and implementations of their contributions. I have already put a couple of their models into the gallerypages, and there’s much more where that came from.
Complex systems theory is a way of thinking about systems with many interacting parts and functions. It draws on physics and the science of modeling dynamic systems. It’s a trans-disciplinary, quantitative science of everything. It often and increasingly gets applied to social systems, often through the methods of agent-based modeling (ABM). ABM has a long history in computational sociology. More recently, it has made inroads into economics and finance (Axtell and Farmer, 2022). That’s important intellectual territory to win over because, of course, it is vitally important for both private and public interests. Its progress there is gradual but steady. ABM and complex systems methods have no dogma besides mathematical and computational essentials. Their eventual triumph is more or less assured. As I’ve argued, ABM and complex systems theory are thus an exciting frontier for legal theory (Benthall and Strandburg, 2021). For these reasons, one line of my research is involved in developing computational frameworks (i.e. software libraries, mathematical scaffolding) for computational social scientific modeling.
Contextual Integrity (CI) is an ethical theory developed by Helen Nissenbaum. It is especially applicable to questions of the ethics of information technology and computation. Central to the theory is the idea of “appropriate information flow”, or flows of (personal) information which conform with “information norms”. According to CI, information norms are legitimized by a balance of societal values, contextual purposes, and individual ends. The work of the CI ethicist is to wrestle with the alignments and contradictions between these alignments, purposes, and ends to identify the most legitimate norms for a given context,. When the legitimate norms are identified, it is then in principle possible to design and deploy technology in accordance with these norms.
CI is a philosophy grounded in social theory. It has never been robustly quantified and many people think this is impossible to do. I’m not among these people. In fact, much of my work is about trying to quantify or model CI. It should come as no surprise, then, that I now see CI in terms of complexity theory. It has struck me recently that what this amounts to, more or less, is a computational social theory of ethics! This idea is exciting to me, and one day I’ll want to write it down in detail. For now, I have some nice diagrams and notes from a recent presentation I wanted to share.
CI is a theory of ethics that is ultimately concerned with the way that values, purposes, and ends legitimize socially understood practices. The ethicist’s job, for CI, is to design legitimate institutions. A problem for the ethicist is that some institutions can be legitimate, but utopian, in that they are not stable behavioral patterns for the sociotechnical system. Complex systems theory, as a descriptive science, is well adapted to modeling systems and identifying the regular behaviors within them, under varying conditions. Borrowing a notion from physics, a system can exhibit many regular behavioral states, which we might call phases. For example, it is well known that water has many different phases, depending on the temperature: ice, the liquid water, steam, etc.
Norms have both descriptive and (ahem) normative dimensions. (This confusing jargon is part of why it’s so hard to make progress in this area.) In other words, for there to be an actually existing norm, it has to be both regular and, to be ethical according to CI, legitimate.
There are critics of CI who argue that one problem with it is that it assumes an apolitical consensus of information norms without addressing how norms might be distorted by, e.g., power in society. This is not terribly fair to Nissenbaum’s broader corpus of work, which certainly acknowledges political complexity (see for example the recent Nissenbaum, 2024). Suffice it to say here that not all individual end up being ‘legitimized’ when ethicists assess things, and that legitimization is always political. Moreover, individual ends and politics can, of course, often be the driver of system behavior away from legitimate institutions. We can’t always have nice things.
Nevertheless, it remains useful to consider how and under what conditions a system could remain legitimate despite technological change. This is what the original CI design heuristic is: a procedure for evaluating what to do when a new technology creates a disruptive change in societal information flows.
Ideally, for CI, when a new technology destabilizes the sociotechnical system’s behavior and threatens it with illegitimate practices, society reacts (through journalism, through ethics, through a political process, through private choices and actions, etc.) and returns the system to a regular behavioral pattern that is legitimate. This might not be the same behavior as the system started with. It might be even better. And that’s OK.
What’s bad, for CI, is if the system gets stuck in an illegitimate but still robust phase.
While there are some applications of CI that serve anodyne ends of parsing and implementing uncontroversial privacy rules, there are other uses of CI as a radical critique of the status quo. This is well exemplified by Ido Sivan-Sevilla et al.’s comments on the FTC ANPR on Commercial Surveillance and Lax Data Security Practices (2022), which is a succinct and to-the-point condemnation of the “notice and consent” practices in commercial surveillance. We live in a world in which standard, even ubiquitous, technology norms depend on “laughable legal fictions” such as the idea that users of web services are legitimate parties to contracts with vendors. It is well documented how these fictions have been enshrined into law by decades of pressure by the technology sector in courts and government (Cohen, 2019).
Together, CI and complex systems theory can show how society can be a winner, or loser, beyond the sum of individual outcomes. There are certainly those that have argued that, essentially, “there is no such thing as society”, and that voluntary, binary transactions between parties are all there is. An anarchic, libertarian, or laissez-faire system certainly serves the individual ends of some, and is to some extent stable until the lords of anarchy create new systems of rules that are in their interest. It is difficult to analyze the social costs of these political changes in terms of “individual harms”, because the true marginal cost is not measurable at the level of the individual, but rather at the level of the phase transition. A complex systems theory allows for this broader view of what is at stake.
This approach also, I think, helps convey the fragility of legitimate institutions. Nothing guarantees legitimacy. Legitimate institutions typically constrain the behavior of some actors in ways that they individually do not enjoy. There are social processes which can steer a system towards a more legitimate phase, but these will meet with resistance, sometimes fail, and can be coopted by bad faith actors serving their own ends.
Indeed, there are those who would say we do not live in a legitimate system and have not lived in one for a long time. “Legitimate for whom?” Even if this is so, CI invites us to have a productive dialog about what legitimacy would entail, by sorting out different motivations and looking at the options for balancing them out. This good faith search for resolutions is often thankless and unrewarded, but certainly we would be worse off without it. On the other hand, arguments about legitimate institutions that are divorced from realistic understandings of sociotechnical processes are easily deployed as propaganda and ideology to cover illegitimate behavior. Ethics requires a science of sociotechnical systems; sociotechnical systems are complex; complex systems theory is a solid foundation for such a science.
References
Axtell, R. L., & Farmer, J. D. (2022). Agent-based modeling in economics and finance: Past, present, and future. Journal of Economic Literature, 1-101.
Benthall, S., & Strandburg, K. J. (2021). Agent-based modeling as a legal theory tool. Frontiers in Physics, 9, 666386.
Cohen, J. E. (2019). Between truth and power. Oxford University Press.
Nissenbaum, H. (2024). AI Safety: A Poisoned Chalice?. IEEE Security & Privacy, 22(2), 94-96.
A recent Dear Colleagues Letter from the National Science Foundation Directorate for Computer and Information Science and Engineering (CISE) calls for proposals for projects to envision research priorities. It is specifically not for research itself, but for promising ways to surface and communicate new R&D directions.
Essentially, the CISE directorate is asking for people to figure out a way to identify the future of computer and information science research. Just, you know, putting it out there.
The CISE Directorate is roughly 38 years old at the time of this writing, and computing and information science have, in that time, transformed pretty much everything.
At the same time, at this present moment, there’s a sense in which computer science feels… saturated. Maybe, indeed, lacking in future vision.
Why do I feel this is so? At least two reasons:
a) In the 90’s and 00’s, so much of the potential of computer science was being discovered and unleashed by startups. Even the companies that are today Big Tech were, then, startups. Now, notoriously, a lot of startups are just weird offshoots of Big Tech companies designed to be absorbed back in when legal or market conditions are favorable. So the technical research agenda is being set by huge companies with in-house research, rather than by a loose network of innovators.
b) “Artificial Intelligence” has for a long time meant “anything that computers can’t do yet”, with the Turing Test as one of the examples of what was still an unsolved problem in computer science. Deep learning has been blasting these unsolved problems out out of the water for almost a decade now. I’d argue that the newish LLM-powered chatbots appear so ominously to be a form of “general” AI is because they command natural language so convincingly — the key challenge of the Turing Test. So, computer science is running out of unsolved problems.
c) At the same time, this so widely hyped and lauded generation of AI, which has been credited with potentially literally apocalyptic powers, has gotten over the hump of the Gartner hype cycle, and it still can’t get hands right. On the other hand, it is supposed to be making software engineering obsolete as a profession, which would in principle cut down on the demand for computer science research.
d) It is now very clear that the success of computing and information science basic research depends on its uptake in commercial and industrial settings, and that these economics depend on business, legal, and social logic that is outside the scope of computer science research per se. Computer and information science research is not successful in virtue of, but rather in spite of, its agnosticism about social context. And, increasingly, that social context is being included within the scope of computer and information science.
So, what is to be done?
One answer, which I intend seriously, is imperialism. By this I mean the expansion of computer and information science research into areas beyond its core. Another answer is that it can occupy itself by adapting to critique. I actually think a combination of both the the best answer.
By imperialism, I mean searching for unsolved problems in other sciences, and trying to crack them with computational methods. This has been done already with Go and protein folding. But most problems in the social sciences remain unsolved problems, computationally. There are indeed parts of the social sciences that are opaque to themselves and without the guiding light of computational theory.
By adapting to critique, I mean responding to the now ample critical literature, mainly produced by humanistic scholars (some legal, some STS, etc.) which aims to show the shortcomings of computer science methodology. Indeed, a lot of “information science” today operates at this critical or political level. Humanistic critique tends to stop at the level of anthropological observation.
What is not yet solved is the internalization of these critiques into computational and information theory and methods, which entail advances in the foundations of computational social science.
There are at least three research arenas that I know of which are getting at parts of these problems.
a) The Agent Foundations research agendas (e.g. PIBBSS, Causal Incentives) that have spun out of the AI Safety research communities. This work has come to understand that some foundational advances in what an agent is, in terms of computation and information, is needed to address longtermist AI safety concerns, and perhaps also more pressing problems of AI compliance in the short term. This has quite a bit of funding from Effective Altruist philanthropists.
b) Various computational institutional theory projects that can be found in the vicinity of Metagov. A lot of this is motivated by the idea of the truly self-governing digital community, a long-held Internet dream, one which got an influx of funding and interest from the blockchain boom. That blockchain/crypto flavor has left it, to some, with a funny smell. But some more academic avenues such as the Institutional Grammar Research Initiative have a more based academic stance.
c) Research into the computational foundations of agent-based modeling, such as that led by Michael Wooldridge and Anisoara Calinescu at Oxford University. Part of the interdisciplinary social science mix at the Institute for New Economic Thought, this research vein finds useful computational methods research that pushes the limits of what social systems can be modeled with computers.
The problem with social scientific problems is that they are extremely hard. They can involve multiple agents in intractable situations. Today, we have almost no social systems that are not also sociotechnical systems where the technology is creating complications, so modeling these systems is recursive and perhaps necessarily approximate. To me, these problems remain philosophically tantalizing, when so many issues seem already to be reducible to fundamentals. Maybe this is the direction of the future of computer and information science research.
I am working on AI safety and ethics research again! I’ve contributed to the “Toward Causal Foundations of Safe AGI” sequence on Alignment Forum, and will soon be shifting my research focus back to using agent-based models to improve software accountability. This is an exciting field to work in, in part because there are no shortage of spicy takes by smart people about how dangerous AI is and how important it is to get it right. There’s also more than a little controversy. I want to unpack this controversy and present my own take, which (a) is not one I find expressed by others, and (b) has evolved since I last wrote about it.
Three positions
With many caveats, because many people writing about this topic are much more carefully researched than I am, I think it’s worth sketching a few different positions on why AI is potentially dangerous and what we should do about it.
The first position is that Advanced AI Poses AGlobal Catastrophic Existential Risk (AIX). This is a position made famous by Yudkowksy and Bostrom, and occasionally echoed by scientific luminaries. The original idea is that an autonomous, self-improving AI that is misaligned will grow in power in pursuit of its goals and slay humanity. The argument goes, basically, that since the slaying of humanity is the worst thing that could happen, this is a terribly alarming and so resources should be mustered to make sure that this never happens.
This position has many critics (I wrote a critique once). But that hasn’t stopped a lot of philanthropic activity directed at preventing this existential risk. Partly as a result of that, the theory of AI X-Risk has grown in sophistication from its original versions (I’ll get to this). There is now a lot of interesting research about AI alignment and corrigibility.
The second position is that AI Is (What We Don’t Like About) Capitalism (AIC), and especially “Big Tech” understood as very large, powerful businesses. One of the more fun articulations of this view is by Ted Chiang. A lot of AI policy has this flavor; Meredith Whitaker provided a political economic critique of AI for the AI Now Salon. An important part of this argument is that AI is not, actually, very autonomous. Rather, in its current manifestations, it depends on the cloud computing offerings of a handful of large companies. So AI is not risky to humanity as a whole. Rather, it is risky to those who are not benefiting from it as an industrial process. These critics suggest that this is most people. It is also risky because the idea of AI masks the social and commercial relations that constitute it. As has been said many times, “artificial intelligence is neither artificial nor intelligent”. (I’ve read this very line in recent work by Kate Crawford and Evgeny Morozov, but Googling for it finds this observation in articles going back at least to 2016 if not earlier).
The third position is that AI Offends Liberal Values (AIL). I mean “liberal” very broadly here, in precisely the sense that Jake Goldenfein use in our paper about AI. I mean that AI threatens to be inegalitarian (“unfair”), to upset the democratic process of self-determination (“misinformation”), to be violate individual autonomy (“manipulation”), and so on. These liberal values are core to how Western democracies operate and are tied up with a lot of real legal liabilities. So there’s plenty of commercial and reputational incentives to work on these kinds of problems. So many do.
A lot of the “debate” about AI safety and ethics concerns which of these views — AIX, AIC, or AIL — is either more correct or, more to the point, is more deserving of our scarce resources: our attention; our labor, if we are researchers or practitioners; our philanthropic donations; our political prioritization. Richards et al.’s recent piece in NOEMA is an example: it argues, like many do, that AIX is a distraction from the pressing AIL position.
Where do I stand on these issues?
It’s political
First, I observe that these different positions have different constituencies. AIX has been a popular position with people raising funding from the extremely wealthy. My suspicion is that the extremely wealthy, being humans, quite rationally do not want humanity to be slain by AI, and so this is a way for them to make philanthropic donations without being altruistic. That the AIX has become associated with Effective Altruism is, in this light, perhaps ironic, though of course, from the perspective of saving humanity, we are truly all in it together.
A great deal of thought has gone into how likely an AI X-risk scenario is in fact. But most people will prioritize based on what is more personally salient. Especially given the uncertainty around how truly remote the possibility of AI X-threat is, people are more likely to be motivated by their comparative advantage with respect to other humans. So, AIL is more successful at orienting the priorities of liberal governments and the industrial corporations that thrive within a liberal state, because for these entities what matters is AI’s legitimacy among the body politic and as a part of these institutions. And AIC is more compelling to those who, for whatever reason, empathize with those that are not well rewarded by capitalism, or who are rewarded by their scathing critique of it.
So to some extent these AI debates are the epiphenomenal discourse and signalling of stakeholders occupying different socioeconomic habitus with respect to the phenomenon of AI. Is it possible to put this politics aside?
These three positions are not, actually, mutually exclusive. (What we don’t like about) capitalism may indeed even pose a threat to liberal values and even an existential threat to humanity, and this perhaps this problem should be where we focus more of our scarce resources. There are a number of obstacle to pursuing this line of inquiry:
(a) those that control the preponderance of scarce resources are winners under capitalism and so are going to experience capitalism as legitimizing of liberalism, not as a threat to it (i.e., they will not see AIL as urgent),
(b) it has nothing to do with AI, and all of the component arguments (AIX, AIL, and even AIC) gain prestige because they are about AI, which notionally is what’s creating so much economic value right now, and
(c) the existential risk probability of capitalism is truly remote, because capitalism is driven by human libido; once AI is identified with capitalism the probability of AIX reduces.
(d) there is powerful ideological view that capitalism improves material abundance, promotes liberal values, and the viability of humanity; this view might be right, in which case most of noise about AI safety and ethics in the grand scheme of things is just people complaining or protecting themselves from legal or reputational liability!
On the other hand, it looks like recent work on potential AI X-risk scenarios has been moving away from the unipolar singularity problem and towards problems of failure to coordinate between multiple actors, including the failure to regulate corporate entities as they grow more intelligent. Andrew Critch has written about multi-polar failure as a result of supply chain miscoordination. The Deepmind AGI Safety team seems to think the most likely X-risk scenario involves a failure to co-regulate or adjust becaust of the deception or lack of transparency of some agents as they build to dangerous levels of intelligence.
This is significant. For years, the AIX position has focused on the purely technical aspects of AI and how these might pose a danger. However, now even AIX researchers and advocates are seeing how the worst problems with AI can be due to failures of socioeconomic organization. This means they have much more in common with the AIL and AIC positions.
We need more exact social science
My own personal frustration with the state of the AI safety and ethics debate is that it raises problems that demand both the rigor of the exact sciences (mathematics, computer science, and so on) and which address directly social and economic phenomena that have been, properly speaking, the object of the social sciences. But the social sciences do not seem equipped to address these questions in a serious way, and so we have endless punditry and speculation.
For the past two years I have been working as a National Science Foundation fellow to try to economically model the effects of personal data flows in the economy. This is just one subproblem of the AI safety and ethics gestalt. I can’t say I have succeeded in my original objective, despite having a wide range of methodologies at my disposal, and great collaborators.
Roughly speaking, I’ve been unable to, in two years, successful bridge between three quite different disciplinary camps:
Realist legal scholars and sociologists of technology, who frequently are capable of noticing and putting into words how AI technology interacts with people, how business drives these operations, and so one. But they rarely provide analytic (mathematical) rigor and so it is hard to empirically test their theories or use them in technical design.
Computer scientists, who are extremely rigorous in their designs and validations but most often avoid speculation or theorizing about the social processes that contextualize the use of computation, let alone constitute it. Computer scientists know why artificial intelligence is useful: because it can perform computation that humans cannot perform without it.
Economists, who are the most practiced at modeling economic systems in their complexity but who remain quite bizarrely attached to rational expectations and unbounded rationality as a disciplinary pillar, despite this being known to be nonsense for many decades. The core issue with artificial intelligence, as it is constituted economically, is that it is useful because our human intelligence is limited. This human limitation is precisely what economists are trained to avoid thinking carefully about.
So, in order to properly test and validate the hypotheses raised by realistic observers of AI in society and the economy, there has to be a conversation between two fields that intellectually want nothing to do with each other: computer science and economics.
Of course, I am being somewhat glib. There are people working at the intersection of computer science and economics. There are economics who work on bounded rationality. There are folks doing difficult empirical validation of realistic social and legal theories of the impact of AI. But this is hard work that requires a paradoxical combination of intellectual humility and ambition. To the extent that the outcomes of such research is uncertain, it does not fall easily into any of the political camps of AIX, AIC, or AIL. I am wondering who else is trying to do this work, and how I can work with them.
Standard setting is an essential part of the governance of networked digital infrastructure. The formulation of the Hypertext Transfer Protocol (HTTP), for example, has had ubiquitous impact. Changes in standards can lead to shifts to the distribution of wealth and power. For example, the move from HTTP to HTTPS (HTTP Secure) introduced encryption that prevented many forms of eavesdropping and tampering, which was a boon to privacy and human rights and a loss to Internet service providers, who had profited from observing unencrypted web requests. In other domains, standards setting is the site of geopolitical and economic tussle, as in the 3rd Generation Partnership Project (3GPP), the standards development organization where the 5th generation mobile network standard (5G) was defined. While standard-setting often presents a seemingly impenetrable soup of acronyms, Susan Leigh Star argued that scholars should seriously study “boring things” like infrastructure, for that is where power lies. Standard setting is a target rich environment for multidisciplinary, impactful research into the political economy of technology.
So I celebrate that at the Internet Engineering Task Force 116 meeting in Yokohama, Japan (March 24-30, 2023), the inaugural meeting of the Research and Analysis of Standard-Setting Processes (Proposed) Research Group, otherwise known as RASPRG. While there are several communities that study standards-setting processes, to my knowledge this is the first organized within a standards development organization (SDO) itself. Because the IETF is a particularly open and introspective SDO, this provides RASPRG with a kind of native reflexivity and reciprocity, and ready allies in establishing channels of data access, instrumentation, and establishment of ground truth. It is an extremely promising research area that has already attracted a diverse set of researchers that includes computer scientists, ethnographers, and many in-between, as well as interested members of the IETF community. We’ve already had a number of interesting discussions about research questions and ethics on the mailing list; the legality and ethics of the research are top-of-mind since RASPRG is accountable to the greater IETF community, and includes within it many with a research background in data ethics.
A major focus of RASPRG is deriving insights from the comprehensive open data records of the IETF itself. Two research groups in particular have joined forces through RASPRG. The “Streamlining Social Decision Making for Improved Internet Standards” (sodestream) project, out of University of London and University of Glasgow, has been a well-funded research initiative of computer scientists with deep connections to Internet governance who have been developing tools to improve internet standards setting. Structured somewhat differently, BigBang is an open source research infrastructure project for studying SDOs and other on-line collaborative settings. Both these groups have built data science tools for studying the IETF and other infrastructure governance organizations, and bring this expertise to RASPRG.
I’m excited about these new developments for several reasons. As a technology policy researcher, I am often part of debates about the regulation of platforms and “AI”. However, arguably the network protocol layer is just as important as these ‘application layer’ technologies, and is an under-studied site for impactful research. It is arguably where the most significant privacy-by-design is happening, as network protocols have direct implications for, for example, the behavior of web browsers and other user agents.
I’ve also found the IETF to be an intellectually vibrant community that is curious about the economic, political, and ethical implications of its own work. We have already had open-minded and multidisciplinary conversations about difficult questions regarding demographic representation and organizational involvement, and have seen cooperation towards worthy answers to difficult ethical questions.
On a personal level, I love seeing the maturation of BigBang into something with an infrastructural role. I launched the BigBang project in graduate school with a hairbrained idea that we could build a data science tool to study the sociotechnical process of building data science tools. It failed to earn me my doctoral dissertation, but it was used in the dissertations of others who have become contributors and core developers. Though quite quirky technically, one core developer has described the project to me as a “boundary object” for bringing together different kinds of epistemic communities and imaginaries. So be it. We seem to descending the gradient towards the most fundamental forms of digital infrastructure governance, and it’s my hope that we become embedded there as a source of reflexive insight. It’s been a slow process, but BigBang is an ongoing success raising an expanding universe of questions.
A cornerstone of all computer science research is the analysis of the difficulty of solving problems. As is well known, some problems, like sorting a list of numbers, are relatively easy. Other problems, like the knapsack problem, are hard. Here, “easy” and “hard” are defined by computational complexity classes: the amount of processing time it takes to solve the problem as a function of the size of the input.
Statistics has its own internal understanding of the difficulty of solving problems. When doing statistical inference properly, you cannot do better than your data and the validity of your assumptions (c.f. no free lunch theorem). You cannot solve a high dimensional problem with low dimensional data (c.f. the curse of dimensionality).
“AI”, or machine learning, or data science, in its current form is the combination of statistics and computer science. Serious researchers in either domain know that the problems they are solving are often hard. (Deep learning perhaps has allowed the AI research community to suspend their disbelief for a time.)
Consider two problems:
A: The problem of predicting Y from input data X, such that the decision whose value depends on the accuracy of the estimate of Y can be made well.
A’: The problem of predicting the consequences of deploying the system that solves A in a complex sociotechnical world.
Which problem is harder?
However hard problem A is, A’ will be harder. To solve A, you need training data for X and Y, and sound inference and optimization algorithms. To solve B, you need not only training data for X and Y (in order to understand the behavior of A), but also training data from which to learn the structure of the sociotechnical world in which the system is deployed. This will be much higher dimensional data than those used to solve A’. (Simulating the total system and getting a distribution over its outcomes may also prove to be complex in terms of runtime–more complex than the original optimization problem involved in solving A).
Considering this argument, its clear why the difficulty with computer scientist’s solving AI ethics problems is not their use of abstraction as a disciplinary problem (see Selbst et al. 2019). Rather, it’s because the AI ethics problem (A’) is, for abstractly understandable reasons, much harder than the AI problem (A).
There is a great deal of humanistic discussion of AI ethics coming from law, anthropology, and so on. Qualitative research and humanistic understanding are wonderful in part because they allow for a high-dimensional understanding of their phenomena. But they are not free from the laws of logic; rather, their powers and limitations can be better understood by showing how they fit within the formally understood mathematics of learning (Benthall, 2016). When “interpretevist” researchers write about AI ethics, they are often doing important work of raising awareness about the consequences of technical systems. This is, it must be said, somewhat easier to do after the fact. They are not solving the AI ethics problem as it confronts the technology designer originally. For these, the principles of computer science apply.
One last point: any model of a sociotechnical system, internalized within an AI component of that system, will be yet-another-AI with potentially undesirable social consequences. We have discussed problem A, and also problem A’. But we can equally consider problem, A”, the problem of predicting the consequences of deployed system A’. And A”’, A””, A^(n), on into an infinite regress. It’s an interesting question whether the complexity of the problem leaps or plateaus after multiple applications of this operation.
References
Benthall, S. (2016) The Human is the Data Science. Workshop on Developing a Research Agenda for Human-Centered Data Science. CSCW 2016. (link)
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019, January). Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency (pp. 59-68). ACM.
The relationship between scientific research, higher education, and open source software has evolved considerably over the last several years. Today, it’s fair to say that most industrially relevant “data science” practice now depends on open source software that was originally built for scientific research purposes. This has in turn legitimized that software; universities have now placed using open source data science software libraries in their undergraduate curriculum. In computer science and by very loose extension other hard sciences, releasing a high quality software tool is a recognizable academic contribution. We’ve come a long way.
The social sciences have perhaps been slower to take the software turn for many notable reasons. One major reason for this is the broad and disparate nature of the social sciences. A related reason is the disciplinary incompatibility of many social sciences with computational modeling. Abutting this academic resistance to software-based social research, however, is the wide adoption of industrial methods for managing and learning from social data. Arguably, the main industrial drivers of data science have always been social science applications, albeit those within a narrow range. Human-Computer Interaction, Computer Supported Cooperative Work, Management Science, Operations Research, and other business-applicable fields have flourished in recent years in ways that traditional “social sciences” such as Sociology, Anthropology, and History have not.
Enter the question of Economics, widely known to be the hardest (most quantitative) of the social sciences. If there were ever a social scientific field that could make the transition over onto an effective software stack, it would be Econ. In addition to what is in principle a methodological resonance, there is also the plausible link between efficient research tools and industrial applications.
Indeed, the beginnings of an open source economics field are underway. There’s an Open Source Economics Lab at University of Chicago. There’s a NumFOCUS sponsored non-profit, QuantEcon, supporting basic economics tools and associated with Nobel-prize winner Thomas Sargent. There’s Econ-Ark, a different economics toolkit funded by the Sloan Foundation. There’s the Dolo project, and so on.
In this loose taxonomy of scientific software maturity developed at an NSF-funded workshop on Scientific Software Incubators, these projects range between Stage 1, developed by a single software team for internal use, Stage 2, developed by multiple software teams for internal use, and Stage 3, a self-governing community deliberately supporting a broader community.
These are, it must be said, so far small efforts in the field of economics. One explanation for “Why?” comes from the Charter of the nascent Journal for Open Source Economics (JOSEcon). Summarizing the motivations for the journal described in that charter, there’s a compelling argument for the need for a high impact journal that requires of submissions sound software engineering behind its computational tools.
There are computational and numerical methods in economics research with many benefits:
More expressive than purely analytically tractable models
Ability to support parameter estimation/model fitting
Software development practice among economics researchers is currently weak
Mainly informal code transfer with little effective code reuse
Publication standards are not guaranteeing reproducibility
Lots of reinventing the wheel
Potential of a replicability crisis
The solution is a change in incentive structure
JOSEcon aims to be a high prestige journal that requires better software practices for submissions
A submission includes:
A well-documented software package
Short script or notebook demonstrating functionality
A couple pages of prose of applicability
Could be new research, or a replication of existing research
Submissions are citable for academic credit towards e.g. tenure
At the moment, there seems to be a bit of a chicken-and-egg problem. Software engineering skills are in short supply among economists. So it’s unlikely that a journal that requires sound software practices behind its submissions will quickly become prominent in the field. On the other hand, it’s possible that the infrastructure for general-purpose scientific publishing will accommodate computational research and it will be left to economists to take advantage of it after the way has been prepared ahead of them.
Current proposals may lack conceptual clarity about software engineering and its precise relationship with academic publication. The incentives and needs of the two fields are subtly different in ways besides how academic research values citations. The library and dependency structure of software depends critically on functional modularity. Arguably, research publications are organized around a more narrative structure. The logic of presentation of a research publication is rarely going to fit the most efficient architecture of computational modeling.
All this points to a fascinating intellectual problem at the core of all this: what is the right architecture for computational economics software tools? Is an economic model a functional unit of logic? Or is it a narrative for presentation? Can the logical units be efficiently decomposed and reused?
Let’s assume for a moment that any politically useful language (“freedom”, “liberalism”, “conservatism”, “freedom of speech”, “free markets”, “fake news”, “democracy”, “fascism”, “theocracy”, “radicalism”, “diversity”, etc.) will get coopted by myriad opposed political actors that are either ignorant or uncaring of its original meaning and twisted to reflect only the crudest components of each ideology.
It follows from this assumption that an evaluation of a government based on these terms is going to be fraught to the point of being useless.
To put it another way: the rapidity and multiplicity of framings available for the understanding of politics, and the speed with which framings can assimilate and cleverly reverse each other, makes this entire activity a dizzying distraction from substantive evaluation of the world we live in.
Suppose that nevertheless we are interested in justice, broadly defined as the virtue of good government or well-crafted state.
It’s not going to be helpful to frame this argument, as it has been classically, in the terminology that political ideological battles have been fought in for centuries.
For domestic policy, legal language provides some kind of anchoring of political language. But legal language still accommodates drift (partly by design) and it does not translate well internationally.
It would be better to use an objective, scientific approach for this sort of thing.
That raises the interesting question: if one were to try to measure justice, what would one measure? Assuming one could observe and quantify any relevant mechanism in society, which ones would be the ones worth tracking an optimizing to make society more just?
One of the reasons why it’s important to think about explicitly about downward causation in society is how it interacts with considerations of social and economic justice.
Purely bottom-up effects can seem to have a different social valence than top-down effects.
One example, as noted by David Massad, has to do with segregation in housing. Famously, the Schelling segregation model shows how segregation in housing could be the result of autonomous individual decisions by people with a small preference for being with others like themselves (homophily). But historically in the United States, one factor influencing segregation was redlining, a top-down legal process.
Today, there is no question that there is great inequality in society. But the mechanism behind that inequality is unknown (at least to me, in my current informal investigation of the topic). One explanation, no doubt overly simplified, would be to say that wealth distribution is just a disorganized heavy tail distribution. A more specific account from Piketty would frame the problem as an organized heavy tail distribution based on the feedback effect of the relative difference in rate of return on capital versus labor. Naidu would argue that this difference in the rate of return is due to political agency on the part of capitalists, which would imply a downward causation mechanism from capitalist class interest to individual wealth distributions.
The key thing to note here is that the mere fact of inequality does not give us a lot to distinguish empirically between these competing hypotheses.
It is possible that the specific distribution (i.e cumulative density function) of inequality can shed light on which, if any, of these hypotheses hold. To work this out, we would need to come up with a likelihood function for the probability of the wealth distributions occurring under each hypothesis. Likely the result would be subtle: the difference in the likelihood functions would be about not that but how much inequality results, and whether and in what ways the wealth distribution is stratified.
Of course, another approach would be to collect other data besides the wealth distribution that bears on the problem. But what would that be? The legal record of the tax code, perhaps. But this does not straightforwardly solve our problem. Whatever the laws are and however they have changed, we cannot be sure of their effect on economic outcomes without testing them somehow against the empirical distribution again.
Another challenge to teasing these hypotheses apart is that they are not entirely distinct from each other. A disorganized heavy tail distribution posits a large number of contributing factors. Difference in rate of return on capital may be one important factor. But is it everything? Need it be everything to be an important social scientific theory?
A principled way of going about the problem would be to regress the total distribution against a number of potential factors, including capital returns and income and whatever other factors come to mind. This is the approach naturally taken in data science and machine learning. The result would be the identification of a vector of coefficients that would indicate the relative importance of different factors on total wealth.
Suppose there are 20 such factors, any one of which can be removed with minimal impact on the overall outcome. What then?
Miller and Page (2009) do indeed address “fat tail” distributions explicitly in the same chapter on Emergence discussed in my last post.
However, they do not touch on the possibility that fat tail distributions might be log normal distributions generated by the Central Limit Theorem, as is well-documented by Mitzenmacher (2004).
Instead, they explicitly make a different case. They argue that there are two kinds of complexity:
disorganized complexity, complexity where extreme values balance each other out to create average aggregate behavior according to the Law of Large Numbers and Central Limit Theorem.
organized complexity, where positive and negative feedback can result in extreme outcomes, best characterized by power law or “heavy tail” distributions. Preferential attachment is an example of a feedback based mechanism for generating power law distributions (in the specific case of network degrees).
Indeed, this rough breakdown of possible scientific explanations (the relatively orderly null-hypothesis world of normal distributions, and the chaotic, more accurately rendered world of heavy tail distributions) was the one I had before I started studying complex systems and statistics more seriously in grad school.
Only later did I come to the conclusion that this is a pervasive error, because of the ease with which log normal distributions (which may be “disorganized”) can be confused with power law distributions (which tend to be explained by “organized” processes). I am a bit disappointed that Miller and Page repeat this error, but then again their book is written in 2009. I wonder whether the methodological realization (which I assume I’m not alone in, as I hear it confirmed informally in conversations with smart people sometimes) is relatively recent.
Because this is something so rarely discussed in focus, I think it may be worth pondering exactly why disorganized heavy tail distributions are not favored in the literature. There are several reasons I can think of, which I’ll offer informally here as possibilities or hypotheses.
One reason that I’ve argued for before here is that organized processes are more satisfying as explanations than disorganized processes. Most people are not very good at thinking about probabilities (Tetlock and Gardner (2016) have a great, accessible discussion of why this is the case). So to the extent that the Law of Large Numbers or Central Limit Theorem have true explanatory power, it may not be the kind of explanation most people are willing to entertain. This apparently includes scientists. Rather, a simple explanation in terms of feedback may be the kind of thing that feels like a robust scientific finding, even if there’s something spurious about it when viewed rigorously. (This is related, I think, to arguments about the end of narrative in social science.)
Another reason why disorganized heavy tail distributions may be underutilized as scientific explanations is that it is counter-intuitive that a disorganized process can produce such extreme inequality in outcomes.
This has to do with the key transformation that is the difference between a normal and a log normal distribution. A normal distribution is a bell-shaped distribution one gets when one adds a large number of independent random variables.
The log normal distribution is a heavy tail distribution one gets by multiplying a large number of positively valued independent random variables. While it does have a bell or hump, the top of the bell is not at the arithmetic mean, because the sides of the bell are skewed in size. But this is not necessarily because of the dominance of any particular factor (as would be expected if, for example, a single factor were involved in a positive feedback loop). Rather, it is the mathematical fact of many factors multiplied creating extraordinarily high values which creates the heavy right-hand side of the bell.
One way to put it is that rather than having a “deep” positive feedback loop where a single factor amplifies itself many times over, disorganized heavy tails have “shallow” positive feedback where each of many factors has a single and simultaneous amplifying effect on the impact of all the others. This amplification effect is, like multiplication itself, commutative, which means that no single factor can be considered to be causally prior to the others.
Once again, this defies specificity in an explanation, which may be for some people an explanatory desideratum.
But these extreme values are somehow ones that people demand specific explanations for. This is related, I believe, at the desire for a causal lever with which people can change outcomes, especially their own personal outcomes.
There’s an important political question implicated by all this, which is: why is wealth and power concentrated in the hands of the very few?
One explanation that must be considered is the possibility that society is accumulated history, and over thousands of years an innumerable number of independent factors have affected the distribution of wealth and power. Though rather disorganized, these factors amplify each other multiplicatively, resulting in the distribution that we see today.
The problem with this explanation is that it seems there is little to be done about this state of affairs. A person can effect a handful of the factors that contribute to their own wealth or the wealth of another, but if there are thousands of them then it’s hard to get a grip. One must view the other as simply lucky or unlucky. How can one politically mobilize around that?
References
Miller, John H., and Scott E. Page. Complex adaptive systems: An introduction to computational models of social life. Princeton university press, 2009
Mitzenmacher, Michael. “A brief history of generative models for power law and lognormal distributions.” Internet mathematics 1.2 (2004): 226-251.
Tetlock, Philip E., and Dan Gardner. Superforecasting: The art and science of prediction. Random House, 2016.
Digifesto 