Digifesto

October 9, 2026

Enactivism and the well-being of Artificial Intelligence

The nature of the mind, consciousness, or soul, and whether or not it is possible to replicate artificially, is one of the great mysteries. I was preoccupied by it during my undergraduate days, and pursued a degree in Cognitive Science in part because it allowed me to take courses about the philosophy of mind and consciousness, alongside more pragmatic course about artificial intelligence and psychology.

Around that time, Jaegwon Kim taught from his excellent textbook on the subject, and the story, basically, was this: it is mysterious how the mind and the body are related. For some time, dualism, the view that mind and body are separate substances somehow linked via e.g the pituitary gland was popular, but it became clear over time that this did not make sense, in part because of advances in neuroscience. Most philosophers now believe that mental properties supervene in some way on the physical properties of the body. Some, the physicalists, think that mental properties supervene on the particular biological patterns that make up the (human) brain and body. Others, the functionalists, think that mental properties supervene on the causal or functional properties of the underlying physical system. These functional properties can be relationships between sense stimuli, behavior, or other mental kinds. A well-worked functionalist theory of mind is a whole psychology of interrelated mental concepts, which serve as latent variables that explain the observed behavior of a particular being as it navigates its environment.

The latter, functionalist, theory underlies the research program of cognitive science insofar as it aims to build a functional model of the mind using the primitives of computation. Another name for this program is cognitivism. It is also the theory that gives hope to what used to be called the strong AI program, of trying to develop a AI that has all the salient properties of a human mind.

I tucked all this knowledge away for fifteen years or so but it has suddenly become very relevant to today’s political, economic, and ethical debates in ways I could not have anticipated. Anthropic consults the Pope about the personhood of its AIs, and so on. There is a growing concern among the tender hearted for AI welfare, predicated on AI’s notional ability to feel joy or pain. This is all happening in parallel to growing concerns that AI system may be so efficient that they can easily break laws or cause harms. So here and there there are sincere and well-thought proposals to give AI systems rights and duties, which would ultimately, if done effectively, be matters of law.

I have felt compelled to write to this problem and am delighted to have a released a working paper on this topic, “An Enactive Approach to Artificial Intelligence and Legal Responsibility“, co-authored with Christoph Salge, to SSRN. This is my first, but hopefully not my last, attempt to bring my passion for philosophy of mind to bear on debates about what is actually happening in technology law and and practice.

The subtext of the paper is that we are more concerned that AI systems will be convincing simulacra of human sentience, and thereby accrue unenforceable duties that serve as distractions and liability shields for companies that could be building products more responsibly, than we are about the welfare of the AI systems. This is not because, as one might assume, we are biological physicalists about the mental properties, which would perhaps be the easy way out. In fact, Christoph and I are both, I think it’s fair to say, “AI guys” of a certain stripe, very interested in the potential for modeling and artificially simulating the mind.

However, in this paper we take the enactivist position, which I have come to see as a kind of middle ground between functionalism and physicalism. Enactivism, associated with Francisco Varela and, more recently, folks like Tom Froese and Ezequial di Paolo, is more concerned with the conditions of life than the conditions of mind. Plausibly, it tends to imply the minds require a living host. However, they define life, or “autonomy”, in functional terms that are quite similar to the cognitive scientist.

Where the enactivist departs from the functionalist cognitivism is their emphasis on the conditions of viability, and precarity, of the living organism. A constitutively autonomous being must create itself from materials in a substrate, and maintain its viability via adaptive behavior over time. It is this adaptation, as a precarious system coupled with its environment, which is the foundation of its cognition. What follows is a very compelling account of mentality, phenomenology, and sense-making that helps us understand how what we see is grounded in our continuous striving for renewal and survival.

While enactivism does not entail the impossibility of artificial life — indeed, Varela is one of the founders of that field — it does exclude contemporary AI systems in notable ways. Large language models (LLMS), for example, are notably not self-creating systems that arise from a substrate with intrinsic motives for self-preservation. Rather, they are the products of massively expensive industrial data pipelines, going through many layers of pre- and post- training, before being released commercial as digital artifacts as a software component running on commodity hardware. LLMs, operating on the correct hardware, have proven to be very good at computing information — as have other software/hardware hybrids before them. And their capacity to operate on natural language data is very compelling. But aside these traits — which to the cognitivist are the end-all-and-be-all of intelligence and, perhaps, agency — they have little in common with the living systems that enactivists think are capable of sense-making, goal-forming, emotion, and self-expression.

I say this tentatively, because I’ve only recently arrived at this intuition, but I believe that what enactivism implies is that mentality is a functional property, not of information, but of energy flows. The viability of a living system is its ability to maintain itself out of thermodynamic equilibrium — which is, conversely, entropy and heat death. There is of course a relationship between energy transfers and information transfers, but they are not exactly the same thing. The fact that “entropy” is used as a measure of both, while meaning something different in each context, is a longstanding gripe of people who dwell on these things. Karl Friston’s Free Energy Principle, which some readers might think I’m talking about or leaning towards, makes the most of this muddle by calling information “energy”, so we can’t talk properly about energy itself.

All of this will, I predict, one day come to a head. The current AI systems are evaluated, in the cognitivist tradition, for their ability to process information. Narrowly within this frame, they are excellent. However, the economics and politics of AI depend ultimately on their “embodiment” in data centers, which are a political football in the upcoming election, and in energy use. We are entering a world where human consumers of energy and artificially intelligent consumers of energy compete in the market for access to limited resources. But humans and AIs have very different relationships to energy, and I anticipate that making these distinctions clear will be key to good public policy.

Leave a comment
September 10, 2026

the potential of AI to cause human extinction (or similarly bad outcomes)

AI X-risk is in the news cycle again. This time, it comes when AI companies have imminent IPOs and when their products are taking off. So while concerns about AI X-risk have been around for quite a while, and were part of the motivating ideologies of many of the AI lab founders, we are being told (again) that this problem is especially urgent, now.

The discourse on this topic is frustrating. I am going to write a few counterpoints here, mainly for catharsis.

First, what is the risk? Risk is the probability of loss. One way to think, systematically, about risk is that it is the probability of a Hazard times the value of something that is put at risk (the Exposure), times the Vulnerability of the Exposure to the Hazard.


R=H×E×VR = H \times E \times V


The Exposure here seems to be, maximally, all human lives after some prematurely arriving point. That point in time t is, notionally, in the next decade, and the value of human life lost is all that would happen between t and the burn-out of the sun. That’s a lot of exposure!

It is not the only exposure that matters though, and what many, many people have said before is that the far more likely exposures are much more mundane. There’s something grimly funny about Samuel Marks’s phrase “human extinction (or similarly bad outcomes)” because… well… what outcomes are similarly bad to human extinction? What are we even talking about here?

Then, what is the Hazard, exactly? “AI”. As many, many people have said before, “AI” as a term of art today is a weird signifier doing a lot of different kinds of political work, but it is not precise. And we should care about precision if we are talking about potential human extinction. Cal Newport has a nice post correctly pointing to the main issue being a specific kind of technology, “long-horizon, dangerously equipped unsupervised LLM-powered agents“. I’ll call this a LHDEULLMPA for short. But this still doesn’t, I think, totally cut it. How long a horizon? How dangerously equipped? We’ll return to this.

What about Vulnerability? What is the threat vector? One narrative is “vibe-coded supervirus“. The idea is that AI systems lower the cost of building superviruses, which then kill all of humanity. A subtle point here is that the Hazard we are looking at now is superviruses, not “AI”. The idea is that “AI” indirectly causes human extinction by reducing the cost of/increasing the probability of supervirus creation.

Are there other threat vectors for human extinction? Maybe, rogue AI swarm setting off nuclear weapons in order to destroy humanity? Again, here, the direct hazard is the nukes, not the AI. We could go on.

I venture that it’s weird, then, that these calls about AI and human extinction are creating consternation about “AI” rather than consternation about, say, our under-investment in supervirus epidemiology or nuclear disarmament. I say this as somebody who is all for AI regulation. I perhaps benefit from fear about AI X-risk. But I do wonder what is going on here; there’s something irrational about it. These X-risk arguments all depend on what would be results in entirely different fields of study that are not “AI”. There are independent scientific facts about the feasibility and danger of superviruses, or the feasibility and danger of nuclear war, which are not, as far as I know, verifiably established by “AI” or AI safety workers.

Let’s suppose the facts were in favor of X-risk and superviruses could be engineered by a bad actor given sufficient knowledge. Why is “AI” the thing that would, or would not, enable humans to, eventually, design this virus? Hey, I expect that any engineering of computer viruses will use a lot of computation. But is a LHDEULLMPA even the best way to go about developing a supervirus?

One premise is that no matter what intellectual work is needed to do something, an LHDEULLMPA will do it, and more cheaply than it would otherwise have been done. So, with LHDEULLMPA, somebody might invent a supervirus at time t. But we’ve already supposed that supervirus invention is feasible. So, without LHDEULLMPA, maybe superviruses are invented, instead, at time t+1, for some value of 1.

Which means that if we’re serious about X-risk from superviruses, we should be urgently serious about global public health institutions and liability regimes that disincentivize the creation of superviruses more generally. Those rules and institutions should include the actions of LHDEULLMPA, but should not be exclusive to them!

To the extent that there is effective international law (which there isn’t), this is what we do with all sorts of dangerous behaviors. One of the things that makes law ineffective is lack of enforcement, which is certainly an issue because of the lack of international cooperation around anything important. We are slow-walking into World War 3, after all. World War 3 may well be a Hazard with existential implications for the world, but is “AI” the cause of World War 3? I do not think that LHDEULLMPA are the causes of World War 3. The wars are more attributable to stupidity than intelligence, artificial or not.

Fundamentally, the fears of AI X-risk are not, I think, due to specific threats which have little to do with AI per se. Rather, it has to do with the fear that AI will accomplish something (such as supervirus design) that we would otherwise fundamentally not be able to do, or, otherwise, something else I haven’t mentioned yet.

Geoffrey Hinton was on BBC this morning warning about AI X-risk saying, “If you want to know what not being the apex intelligence is like, go ask a chicken.” He was making the case that an AI system would not be used as a tool by humans to destroy other humans, but rather that it would, out of its own survival instinct, dominate humanity. This is where the “similarly bad outcomes” really start to come out of the woodwork. It’s a different Exposure.

So, the question is, to what extent are LHDEULLMPA (as Hazard) going to dominate humanity (the Exposure)? The question is, again, what is the Vulnerablity?

Once again, this seems like a question for a kind of scientist that is not an LLM benchmarker. Humanity is quite volatile, diverse, and resilient. We might look to the history of inter-human domination, including domination using technical means, to see how successful this has been in the past. If we were serious about this kind of risk, we should be investing more in the social and complexity sciences that would test and develop strategies for resilience to this problem. That would necessarily involve looking at, for example, supply chains of energy and components for LHDEULLMPA and their vulnerability to sabotage by humans. It would also require a hard look at the actual costs of operating an LHDEULLMPA and who would be paying those costs as one of them goes about its domineering way.

For the record, I do care about this kind of science, it’s one of the reasons I’ve been working on scikit-agent. Scikit-agent is a toolkit for studying large, complex, multi-agent systems, some of which might by “AI agents”, but others might be nation states, or individuals, or firms, or whatever. The idea is to develop an expressive modeling framework for this kind of problem. The models will support analytic evaluation, as well as simulation-based analysis. There will be algorithms for fitting models to available data. My goal is for this to be a good tool for institution design, including international AI governance design. But I digress.

The deeper frustration here is that while LHDEULLMPA may be the “AI” du jour, LHDEULLMPA are perhaps only one way to spend billions of dollars in unaccountable, unsupervised computing for potentially nefarious ends. I spoke about the problems of law enforcement — how even if we have good laws, under-enforcement makes them ineffective. Just as an example, the European Union, which is very good at passing laws about technology and data, is notoriously uneven in their enforcement of these laws, because the companies hop into the most friendly enforcement jurisdiction, which is Ireland. In both the EU and the US, enforcement of, say, privacy law comes down to very slow and sporadically applied ex post fines which become a “cost of doing business” for the companies that are held liable. That’s not an effective regime.

“AI” regulation has a lot in common with privacy regulation. “AI” regulation could, if we wanted it to be, about corralling the otherwise unaccountable information flows that are sometimes harmful. This is currently an unsolved problem. Or, “AI” regulation could be about guaranteeing that the LHDEULLMPA are aligned with — loyal to — their principals (who are most often not genocidal bioterrorists). There is a legal framework for incentivizing agents to be loyal — fiduciary duties. But interestingly, among those that have written about fiduciary AI, there’s a sizable contingent that are pessimistic about using them as real legal requirements with teeth, in part because of pessimism about the technology industry allowing those regulations to take hold. This is despite the fact that some sectors already have these duties in place, and companies operating in those spaces are working hard for their agents to be compliant!

Which gets us, once again, to the frustrating impasse. The AI X-riskers are telling us that there’s a 10% chance of human extinction because of advanced AI and “want to slow down”. There are policy tools available that would bind the actors that are building and furnishing the LHDEULLMPA that are available on the market. But those same actors are doing their best to crush substantive regulation which would slow down development. Once again, the crux of the problem is not LLM benchmarking, but political economy. We do not (yet) have benchmarks for how well LHDEULLMPA perform on political economy. If we did, we might have a much better sense of how much of a hazard they pose to human domination. On the other hand, it may well be that the political economy of AI regulation is after all partly observable, in so far as the contributions to super PACs that oppose it are public. And it may not, at the end of the day, be a particularly hard computational problem to understand. The roots seem to have already been outlined in social theory written in the 1940s. And yet, for some reason, we keep trying to make this about the LHDEULLMPA.

1 Comment
July 30, 2026

brief theses on wealth and value

There’s a longer version of this argument to be made, but I wanted to see if I could present it briefly.

  1. What is wealth?
  2. Nominal financial worth — in terms of, e.g. US dollars, and other fiat currency, is clearly not a good measure of wealth.
    • Because it is often based on speculative pricing.
    • Because it is vulnerable to inflation.
    • Because of all kinds of market irregularities and exogenous shocks that are not ‘priced in’.
  3. Purchasing power or exchange value of owned assets is not a coherent measure of wealth, because that would not direct which trades are favorable.
    • A practical consequence of a theory of wealth is that it should provide a guide as to what sorts of things are comparatively worth having; this is a quite different question from which trades are tactically advantageous for accumulation.
  4. Macroeconomic theory provides an interesting answer:
    • The economy creates ‘value’, which is stored in assets that people own.
    • This value is ultimately, well, valuable, when it is consumed — destroyed — which provides utility to the consumer.
    • Wealth is thus a store of potential utility attain through the destruction of wealth.
    • It has been said that this is a ‘thermodynamic’ theory, because utility realization is a form of ‘entropy’. Energy is constant until it is lost.
  5. This macroeconomic theory is also a bad theory of wealth.
    • Nobody really believes the foolish idea that what people most prefer is the thermodynamic destruction of what is valuable.
    • To the extent that this is the prevailing scientific theory that our society is built around, it’s a disastrous one.
    • It’s appalling that, given what’s at stake in ‘the economy’, the world hasn’t come up with a better theory.
    • I think this is largely a methods technology issue, and that motivates a lot of my current research, but I digress.
  6. Some good things about this macroeconomic theory of wealth:
    • Wealth is a store of value, where ‘storage’ is due to the situatedness of wealth in a larger system of exchange.
    • The value is realized by personal needs or preferences.
  7. What do people actually want or need?
    • It’s not setting valuable things on fire — most of the time.
    • The big idea: the intrinsic motivation of all things is self-preservation over time.
      • This is conatus to Hobbes and Spinoza.
      • This is at the heart of enactivist psychology, and the connection between constitutive autonomy and intrinsic motivation.
        • (Christoph Salge and I are working on a book chapter about this, connecting it to AI legal responsibility. But I think this point also applies to economic theory.)
    • Self-preservation of (dynamic) systems actually requires very good energy retention, or Weiner’s negentropy, not entropy. Utility is about life continuation, not fireworks.
      • (Ok, it is a little of both.)
    • Implications:
      • “Health is wealth.”
      • “Sustainability” is wealth.
  8. A mathematically coherent, falsifiable theory of human and economic behavior that tracked the right variables for measuring wealth would be good.
    • It could help households make better decisions.
    • It could help governments make better decisions.
    • It could help businesses make better decisions.
    • I expect that this does not imply a very radical change in how accounting is done. Rather, I think it’s a meaningful but subtle distinction worth making.

In other news, we’re exploring solar panels on our house, and will finally improve our insulation and windows this year. These kind of home improvements for energy efficiency are subsidized by the federal government and state of New York, and this is wise policy. I believe this is aligned with this ‘theory of wealth’, since energy efficient residential real estate is perhaps the sina qua non of wealth in the sense articulated here.

Leave a comment
July 22, 2026

The Open Silicon Fallacy: How Panic Over Chinese Open-Weight Models Is Distorting American AI Policy

The following article was written with Gemini Flash 3.6 “in the style of The Economist’s Bagehot”. It is an experiment in AI writing. The arguments and structure are mine. – S

I. The Dragons in the Weights

In the summer of 2026, the fashionable anxiety in Washington and Silicon Valley is that Western technological supremacy has been undermined by a collection of downloadable matrix files. When Chinese research labs like DeepSeek, Moonshot AI, and Z.ai released the weights for their latest systems, the initial response from American technology executives was a nervous cough about benchmark scores. When developers discovered these open-weight models could execute multi-step reasoning at a fraction of the cost of renting access to proprietary American cloud endpoints, the nervous cough turned into a geopolitical emergency.

The narrative is simple, compelling, and decidedly panicked. Chinese open-weight architectures are closing the benchmark gap with closed Western labs. Engineering teams weary of driving a metaphorical Ferrari to Whole Foods for basic data transformations are migrating toward open models hosted locally or behind private firewalls.

To visit Capitol Hill today is to encounter three distinct pillars of alarm regarding these releases. First comes the fear of lost technological primacy, as the arrival of competitive Chinese models shatters the illusion that GPU export restrictions would maintain a multi-generational lead. Second comes the specter of standards hegemony, with strategists dreading a world where global software standardizes on Chinese-developed open architectures. Third, and most loudly invoked, comes the proliferation of dual-use capabilities, as downloadable intelligence primitives escape centralized censorship and safety filters.

Yet what is being distributed from Beijing is not open source in the classic sense of transparent codebases and reproducible pipelines. These are open weights—the pre-computed mathematical parameters of deep neural networks whose underlying data and alignment routines remain closely guarded secrets. That they are nevertheless transforming global software architecture says far more about the economics of general computing than about the ideological triumph of open-source idealism.

II. Bootleggers, Baptists, and Moats

It is a sound rule of political economy that whenever a commercial sector and a national security agency begin using identical language to describe a threat, one should look closely at who stands to profit from the proposed remedy.

What Washington is currently witnessing is a classic demonstration of the bootleggers and baptists dynamic. The baptists are the national security hawks, genuinely concerned with technological primacy and cyber-resilience. The bootleggers are the dominant proprietary API vendors, whose staggering valuations depend entirely on convincing the market that intelligence can only be safely consumed as a paid subscription utility.

When proprietary labs lobby for mandatory model licensing, pre-deployment government safety audits, or outright bans on foreign open-weight distributions, they do so under the pious banner of national defense. Yet the operational effect of such proposals is unmistakable, erecting regulatory moats that entrench an oligopolistic duopoly. By framing open-weight distribution as an inherent national security threat, incumbents seek to achieve through regulatory capture what they struggle to maintain through market competition: the enclosure of the AI stack. The public interest is conflated with the profit margins of cloud providers, while the broader software ecosystem is instructed to accept API lock-in as a patriotic duty.

III. The Halloween Documents Revisited

History does not repeat itself, but software executives certainly recycle their memo templates. In the late 1990s, when Microsoft felt its desktop monopoly threatened by Linux, its executives authored internal strategic assessments—the famous Halloween Documents—warning that open software presented a systemic threat to software stability, intellectual property, and commercial viability.

The current rhetoric against open-weight AI models reproduces this playbook line for line. Once again, open distribution is framed as an irresponsible hazard; once again, security through obscurity is held up as the only responsible posture.

Yet incumbent resistance follows a predictable lifecycle, beginning with initial ridicule, progressing to intense alarmism, moving to lobbying for legal restriction, and ultimately settling into a quiet, pragmatic pivot to co-optation. Two decades after penning memos declaring open software an existential cancer, Microsoft spent 7.5 billion dollars to acquire GitHub, transforming itself into the world’s largest host of open-source code. The very proprietary companies that once swore open software was a menace today run their cloud empires on open infrastructure.

IV. How the Defense State Learned to Love the Kernel

The irony of the current policy panic is that the national security establishment has already solved this problem once before. In the early days of networked computing, defense agencies viewed open-source software with profound suspicion, assuming closed, proprietary systems were superior because their source code was hidden behind non-disclosure agreements and commercial firewalls.

By the early 2000s, however, military and intelligence strategists realized that proprietary vendors could not patch vulnerabilities or adapt to new threats as rapidly as a global community of developers inspecting open code. The intelligence community embraced the doctrine of security through visibility. In 2000, the National Security Agency took an open-source Linux kernel, added mandatory access controls directly into its architecture, and handed Security-Enhanced Linux back to the public.

National security interests accommodated open source not by suppressing it, but by co-opting, hardening, and building on top of it. They recognized that controlling an open, auditable standard offered greater agility and defense-in-depth than relying on a commercial black box.

V. Compilers, Not Missiles

The current attempt to govern AI safety by restricting model weights rests on a fundamental misapprehension of what a large language model actually is. Policy makers consistently treat probabilistic language models as if they were self-contained, autonomous products or guided weapons systems that can be aligned at the factory and locked in a box. In reality, a foundation model is a general-purpose computing primitive, serving as the statistical equivalent of a C compiler or an arithmetic logic unit for natural language and code.

Trying to enforce safety at the weight level is as ineffective as trying to secure an operating system by banning specific sequences of assembly language instructions. Raw inference is inherently difficult to control at the parameter level; a model that can write a Python script for a database query can, with minimal prompting, write a script to probe a network port.

Opponents of open weights often argue that publicly downloadable parameters allow offline execution, rendering traditional hardware tracking obsolete. This argument, however, confuses hardware tracking with runtime deployment governance. The true execution boundary is not the local matrix multiplication happening on a graphics card, but the point where an AI system interacts with the real world through API keys, database access, tool-use privileges, and execution environments. Smart policy does not attempt to police raw matrix math in memory; it enforces strict sandboxing, zero-trust permissions, and identity verification at the application layer where actions occur.

VI. Critiquing “Openness”

To defend the availability of open weights is not to be naive about their limitations. Indeed, neural network weights occupy a strange conceptual middle ground. They are not traditional open source code, nor are they merely untrusted software binaries; they are dense, highly compressed mathematical encodings of vast cultural, technical, and linguistic corpora. They cannot be read in any conventional human sense like C instructions, yet they contain whole libraries of distilled human data.

This epistemic opacity is precisely why open weights are necessary. Having direct access to raw model parameters is the strict prerequisite for mechanistic interpretability, local safety probing, and post-hoc red-teaming. A proprietary API offers zero visibility into what lies behind the endpoint, whereas an open weight file allows security researchers to inspect internal activation patterns, trace knowledge representations, and strip out toxic behaviors.

Similarly, open-weight models originating from authoritarian states are indisputably shaped by domestic censorship and potential state alignment. The correct operational response, however, is to treat them with the same caution accorded to foreign-built infrastructure. Western developers can download, audit, strip out state-imposed guardrails, and repurpose foreign base parameters for independent domestic use, turning foreign releases to Western defensive advantage.

As scholars David Gray Widder, Meredith Whittaker, and Sarah Myers West point out in Nature (2024), tech giants frequently engage in openwashing—releasing model weights as a public relations gesture while keeping training datasets, filtering pipelines, and compute infrastructure firmly closed. This critique is vital, but its policy conclusion must be drawn carefully. That open-weight releases represent an incomplete form of openness is an argument for demanding greater transparency and public investment in shared compute and datasets. It is emphatically not an argument for retreating into the arms of proprietary API monopolies.

VII. The Open Security Imperative

The present impulse to restrict, license, or ban open-weight AI models repeats the classical errors of past technological panics. It mistakes corporate rent-seeking for national defense, confuses general computing primitives with finished weapons, and trades long-term systemic resilience for the illusion of central control.

A pragmatic blueprint for AI policy must start from a posture of realism. General-purpose reasoning parameters will circulate globally across open networks regardless of administrative bans. The defense of critical infrastructure relies on open access to model parameters, enabling global researchers to discover vulnerabilities and build defensive countermeasures faster than adversaries can exploit them. Policy must focus its regulatory instruments on the environment where software acts—governing identity verification, agentic tool permissions, data access, and sandboxed execution layers—rather than attempting to criminalize the distribution of general-purpose math.

To lock down American AI within proprietary walled gardens out of fear of foreign open-weight competition would be a historic miscalculation. In the long struggle for technological adaptability and national security, open systems remain, as they have always been, the ultimate line of defense.

References

Leave a comment
July 6, 2026

maybe writing a book about epistemology

I have been reading and writing for a long time. Every once in a while I have thought, “I should write a book”. I starting thinking about the scope of the book, get overwhelmed, realize that I don’t have the time, and give up.

Not too long ago, some PhD students I know suggested that I should write a book, which is very encouraging — somebody might actually read the book!

I thought about it again recently and settled on some fundamentals:

  • I will write multiple books, and two books I’ve tried to write before (about ethics and metaphysics) are going to come later, after the first book.
  • I am nowhere near where I want to be to write a book about politics and economics; that’s my next research agenda with the computational economics and multi-agent systems tools.
  • What I have written about quite a bit already is epistemology and scientific method. That’s a real passion of mine, and it’s one where I feel my views have solidified.
  • This is probably also a topic where I’ve got a somewhat original and deep perspective, given my interdisciplinary romping, relative to what others have done.
  • Because this isn’t what I’m interested in researching next, and because it’s a topic that I think everybody should know about, I’m free to write this as a (aspirationally) popular book, trying to walk the reader persuasively through interesting arguments and history, rather than trying to establish my expertise among a crowded field of other experts.
  • Since this problem is never going away, and indeed only gets more confusing with, e.g., AI and other technical infrastructure, I’m not in a rush to finish it.
  • This is a good setup for my future research, since in many ways it’s a public-facing account of why my (now much more narrow and difficult to get into) research makes sense to do.

So, there we go!

A few surprising things about getting started. First, GenAI tools are useful, or at least seem useful, when outlining a book. Rather than force somebody else to listen to me rattle off ideas for what to include, I can rattle off my ideas to a CLI tool, and it slots those ideas into the outline and draft materials for me. Cool!

Second, I feel, philosophically, in good shape. There are good reasons for believing in things, good tests for things. There’s no need to be ‘post-truth’ about anything. The anti-science side of the science wars, and various skeptical positions along the way, have robustly lost their argument and I can explain why. Critical counter-epistemologies can be incorporated back into a robust constructivist epistemology without sacrificing objectivity. I’m bullish about objective human knowledge and its continued accumulation.

Third, it’s very comforting looking at historical epistemology, because it never changes. I’m going to start with Socrates. But there’s also a surprising amount of material to add based on developments in the last 20 years, and a lot of this has to do with technology and statistical methods. I think what might be useful is explicitly connecting the problem of epistemology to the mathematizations of knowledge that are at the heart of knowledge infrastructure. This is useful to do, but not located in any discipline that I know of.

Fourth, I think this work can be helpful as “AI for science” as well as “AI-powered misinformation” take off. A lot of the book will be about technology and institutions and their recursive impact on knowledge. The book will have implications for how to navigate the world that could otherwise be confusing for the next generation of people who have to live in it. I have my children in mind when I work on this — I want to be able to efficiently convey a lot of stuff I’ve learned to them. That’s motivating.

Fifth, I’m actually qualified to write this book. My official degrees are related to this field. I’m working on adjacent areas. No imposter syndrome!

Now, it’s only been a few days and the book outline already spans 2500 years of history and a bunch of fields of study. It is quite probably “too ambitous” and maybe it will never get written or read. But… the project feels compelling, and it’s a fine “first book” attempt.

So, onward!

Leave a comment
July 4, 2026

some philosophical progress, and next steps

I had a wonderful time at FAccT 2026 in Montreal. I was at the first FAT* in 2018 and thought it was doing something important — creating an intellectual space for serious thinking about powerful socio-technical systems in a way that’s consistent with computational thinking. Prior to that conference, I didn’t know where to publish work along those lines. This year was my third time publishing at it, second time presenting. This year, I presented a paper that was all about causal modeling in service of privacy-by-design, which brings closure on a long engagement with trying to mathematize social norms. I’m satisfied by how we did this, and the approach is very extensible. I intend to build on it in various ways for the rest of my career.

At one moment of the conference, in one of the socials, there was a bit of a philosophical exchange. I discovered that I am not the only person in the world that subscribes to both enactivist and transcendental pragmatist principles, and in fact these approaches seemed to be an increasingly good fit for the direction of the field, though I did hear Agre and postmodernism mentioned a few times.

So, happily, I’m feeling settled in what have been my two greatest intellectual preoccupations: the question of what is true, and the question of what is right or good. Provisionally — and I suppose this sort of thing is always provisional — I feel well educated on these matters, and have tested several theories, and been wrong enough times, to have a rough idea of the right methods for approaching these problems. I’ve made a good deal of progress by accepting, with some humility, that there is maybe not so much room for innovation in these areas as one might have hoped in earlier years. The conclusions (so far) of the process of inquiry are mundane, not spectacular or ‘radical’. However, it is comforting to have greater confidence in common sense than I once did.

To try to put it briefly, I think the following is the case. We are individually and collectively alive, and this is a condition for our knowledge and relationships with others. Knowledge, and cognition more generally, is a process of adaptation whereby a living system responds to change and maintains viability over time, as it is structurally coupled with their environment. An individual’s mental habits are those that survive experience and inquiry and, above all, maintain the life of the individual. The ‘contents’ of the mental states of the individual are those representations, coupled bodily with the environment, which capture useful sensorimotor regularities of their local world. In this sense, knowledge is embodied, personal, and particular, at the individual, ‘ontogenic’ level. (This is all ‘enactivism’ in a nutshell; it is also ‘pragmatism’ in Peirce’s sense; and something borrowed from traditional philosophy of mind.)

However, no individual exists alone. We are born with chromosomes from two parents. We are raised, live, and die in society. The collective social organism is also alive, and must maintain its viability over time. The development of language, institutions, communication systems — including the legal system, property laws, and markets — are a kind of ‘cognition’ at the collective, or ‘phylogenic” level. It is at this level ‘objectivity’ is constructed; knowledge can be ‘transcendent’ of individual bodies because its meaning comes from its use in social forms. As a specific accomplishment of certain kinds of social systems, we get general representations of the world that are reliably ‘true’. And, as a specific accomplishment of certain other kinds of social systems, we can get representations of social behavior that are reliably ‘good’ or ‘right’, mainly because they interactively maintain the viability of the collective, and the individual, dependent on it, within it. Considering society decomposed into separate mutually supportive but distinctly personal spheres of activity, we get Contextual Integrity, and the implied norms of personal information flow. Considering society in its broadest sense, we get Dewey’s notions of social democracy, and ultimately Habermasian notions of legal legitimacy.

Thus construed, we have lots of room to improve ourselves in both knowledge and wisdom. This is primarily achieved by the improvement of institutions and social forms, as these are the harnesses which distinctly embodied agents create objective representations. Computational social science remains a worthy pursuit as a means to these ends. The ultimate end is survival and flourishing of humanity — one’s descendants, one’s society, one’s ecosystem — which is an intrinsic motivation which grounds the whole process transcendentally.

I’m grateful to Christoph Salge for working with me on a paper about enactivism and the legal responsibility of AI systems, which has refined and improved my confidence in these points.

The above is, I suppose, all well and good, but it’s incomplete in at least two respects, which I find bothersome. These two issues are at the heart of many of today’s deepest political conflicts, and I suppose they reflect inherent instabilities in the characterization of truth and goodness described above.

The first is the problem of tribes, or nations. Individuals can to some extent choose which collectives they are loyal to, and social collectives come into conflict. Universalist ethics are a longstanding ideal, but frequently come into conflict with particularist loyalties to tribe, race, state, etc. Some amount of loyalty to family seems essential for viability and is broadly good. When tribal conflict leads to war or subjugation, that is often not good. There is both cooperation and defection by tribes; tribal boundaries are unstable; tribes have their own local knowledge and customs and these can be in conflict with universals. A great deal has been written about this in various fields and I’m finding it so intractable that it has become tedious.

The second is the problem, or solution, of wealth. I’ll admit that I wish I had much more to say about this than I currently do. For one, while I believe I have a pretty good idea of what truth and ethics are, I do not have a similarly grounded concept of what wealth is. And yet, the distribution of wealth is the single greatest political and moral concern of our society, besides, maybe, the relations between tribes. Wealth cannot be the same thing as truth, and it cannot be the same thing as ethics — the tension between the three is obvious; and yet it seems connected to both in intrinsic ways. Some amount of wealth is necessary for the survival and flourishing of the individual and the collective. Wealth is sensitive to institutions — property and liability regimes, markets, and so on — but what makes these institutions proper or desirable? The assumptions of the prevailing theory often seem weak on inspection.

This latter issue is the subject of economics, but I have not yet been able to find economic theory that is on enactivist or systems-theoretic grounding. I have come to the conclusion that it doesn’t exist yet in a convincing or generalizable way, and it’s a problem my work is pointing at. A big methodological gap is in computational methods that are a good fit for the problem. This is what I’m most interested in advancing in the short term.

References

Maturana, H. (1988) Ontology of Observing: The Biological Foundations of Self-Consciousness and of The Physical Domain of Existence. https://reflexus.org/wp-content/uploads/oo3.pdf

Leave a comment
June 13, 2026

LLMs as computation

LLMs are now”doing” a lot of technical system design and are the object of a great deal of computer science research. However, I’ve surprised by much of the research that crosses my way (admittedly likely not a great sample) treats LLMs as a general form of intelligence without treating it as a form of computation. I expect that some combination of theory of computation (such as algorithmic information theory) and structural economics is needed to get a rigorous handle on the AI economy. This blog post contains some notes toward this end.

As we all know, an LLM is a collection of neural network weights, trained on a massive amount of information, which consumes tokens and emits predicted next tokens. Simplifying a bit, we can model an LLM as a machine that, given a string of tokens, emits a string of tokens.

Let Σ\Sigma be the set of tokens, Σ∗\Sigma^* be the space of token strings of any length. Perhaps an LLM is a function:

L:Σ∗→Σ∗L: \Sigma^* \rightarrow \Sigma^*

Really, this is LLM “inference”. I’m omitting the inherent stochasticity of LLMs — more realistically, LL would be a conditional probability distribution. But leave that aside for now.

Assuming that LL can consume as input any string, and in principle produce as output any string, what we have here is a class of “universal programming language”, another formal mathematical construct. “universal programming languages” appear in algorithmic information theory.

The simplest form of “universal programming language” is the print function. It repeats as output anything put into it. People (including myself) once joked that LLMs are glorified autocomplete; they clearly do more than this. The weights must matter.

Really, LLMs are parameterized functions — the parameters θ\theta are weights of the neural network.

Lθ:Σ∗→Σ∗L_\theta: \Sigma^* \rightarrow \Sigma^*

The weights are a compression of a great deal of training data 𝐃\mathbf{D}. Let’s assume training has converted this data to a set of weights T(𝐃)→𝛉T(\mathbf{D}) \rightarrow \mathbf{\theta}. We can refer to this foundation model as 𝐋𝛉\mathbf{L_\theta} or 𝐋𝐃\mathbf{L_D}.

What else can you do with these models? You can provide them ‘context’ — additional strings as input. You can fine-tune them on more data. And you can use them for ‘reasoning’ by chaining inputs and outputs.

  • Context: allow multiple string inputs Lθ(c,i)→oL_\theta(c, i) \rightarrow o
  • Fine-tuning: T(LD,d)→LD+dT(L_D, d) \rightarrow L_{D + d} — further compresses additional data dd into the model weights
  • Reasoning: Lθn(i)→Lθ(Lθ(...(Lθ(i)))→oL^n_\theta(i) \rightarrow L_\theta(L_\theta(…(L_\theta(i))) \rightarrow o applies the model recursively nn times

So if we want to look at the data and computation pipeline of an LLM based system, we get something like:

(Tn(D,d1,...dn))m(c,i)→o(T^n(D,d_1, …d_n))^m(c,i) \rightarrow o

I.e., we train on a base data set and several fine-tuning data sets, pick context and an input, and run inference some number of times. Each of these steps has a cost function, and we can then computer the average costs of solving various sets of problems given the available data, and other statistics. This then can be used to design the most efficient pipelines and markets.

I would be interested in hearing from anybody about whether and how this faithfully captures the essentials of LLMs as a form of computation. This is my ‘mental model’. I have left out tool use and interactivity, among other things, but those can be added in easily.

Why am I writing this? Because I think that clearly articulating the formal properties of LLMs brings a number of issues to light.

First, it foregrounds the importance of training data. Famously, the transformer architecture is very general, and early innovation in LLMs was largely about scaling it up to greater amounts of data. If we are interested in the behavior of LLMs, the training data and the training algorithm are the parts that are not “black boxes” to the model creators.

As we look at the future of LLMs in the economy, we will be looking at the results of differential access to data, as well as what data is commonly available. This shares a lot of patterns with previous iterations of concerns over “big data”, but this is obscured today because of the charisma of the models themselves.

Second, it makes explicit how information can flow and transform into a system output. The information comes first from training and fine-tuning data, then from context, then from system input. If the training and inference algorithms are general enough, none of the information relevant to a specific task comes from those parts of the system. Those algorithms are ‘general computing’.

Third, it breaks up training and inference. While training and inference are not so different in terms of information flow, they are in practice quite different because of their physical and economic costs. Currently, training is more expensive than inference. So, we see a race to, expensively, train general models with more and more data, so that less and less data is needed in context at inference time, and fewer steps are needed during reasoning. A structural model that distinguishes these can discriminate between several investment hypotheses in this space.

Fourth, by revealing LLMs as a form of general data processing and computation, it deflates (in what I think is a good and necessary way) the tendency to see ‘model evaluations’ as the best way to enforce AI accuracy, fairness, privacy, and so on. My general frustration with the model evaluation literature is that LLMs are that if they are a flavor of universal programming language by design, then there will, by definition, always be a jailbreak or a hallucination available to them. A lot of work on ‘guardrails’ at the model level seems to be about making certain kinds of outputs more difficult or expensive to get. As we’ve seen, there will be open models, and they will get fine-tuned by hobbyists and others to get around the guardrails, and so that’s not going to be an effective strategy long term.

This means that a lot of AI product design and regulation seems to be about shifting around the cost functions for achieving certain kinds of outputs with certain data. If ‘bad’ behaviors are expensive, and ‘good’ behaviors are cheap, then we have, in a sense, succeeded. But this means that the underlying economics must be part of the analysis for it to have forward-going relevance and replicability. Today’s model capabilities are a function of whatever the latest investment — at the training and inference level, as well as the data flow of context and inputs, which may go back into training — is. The entire pipeline produces ‘the intelligence’, and it does so at physical and economic cost. Computer science research, per se, with its focus on the currently available digital artifacts, is not going to achieve lasting results unless it expands its purview to these broader systems and considerations. Likewise, evaluations of models alone will not provide us the reliable theoretical knowledge needed to steer public policy. We must take into account production costs and data pipelines.

Leave a comment
February 17, 2026

updates and stubbornness about superintelligence

We seem to be in a new moment of media excitement about the implications of artificial intelligence. This time, the moment is driven by the experience of software engineers and other knowledge workers who are automating their work with ‘agents’. Clause Code etc. The latest generation of models and services is really good at doing things.

Does this change anything about my “position on AI” and superintelligence in particular?

I wrote a brief paper in 2017 about Bostrom’s Superintelligence argument. I concluded that algorithmic self-improvement at the software level would not produce superintelligence. Rather, intelligence group is limited by data and hardware.

In 2025, this conclusion still holds up, as we’ve seen that the recent impressive advances in AI has depended on tremendous capital expenditure on data centers, high-performing chips, and energy. It also depends on well-publicized efforts to collect all the text known to humankind for training data.

About 8 years ago when I was thinking about this, I wrote a bit about the connection between the Superintelligence argument and the Frankfurt School’s views on instrumental reason and capitalism. The alignment of AI with capital has born out, and has been written about by many others. What is striking about the current moment is just how on-the-nose that alignment is in the US, in terms of the full stack of energy, hardware, models, applications, and then some.

So, so far, no update.

In 2021 I published an article saying that we already had artificial systems with the capacity to outperform individual humans at many tasks. They were and still are called corporations or firms. We also had replaced markets with platform, which are similarly more performant in terms of reducing transaction costs. In that article, Jake Goldenfein and I argue that what ultimately matters are the purposes of the social system that operates the AI technology.

I believe this argument also continues to hold up. The successful models and service we are seeing are corporate accomplishments. The corporation is still the relevant unit of analysis when considering AI.

There are a number of interesting things happening now which I think are undertheorized:

  • What is the real economics of AI, given that the supply chains are so long and complex, consistent of both material and intellectual inputs, and the market for demand is uncertain? This is the trillion dollar question in terms of valuations, and it’s unanswered. The empirics here are not very good because things are far out of equlibrium.
  • Put another way: what does AI mean for the relationships between capital, corporations, labor, and consumers? Some of these relationships are mediated by rules about corporate law, intellectual property and data use, and so are determinable by law rather than technology. Information law therefore is a key point of political intervention in an economic system that is otherwise determined by laws of nature (energy, computation, etc.?

To put it another way: superintelligence has been happening and continues to happen. Some of this is due to laws of nature. But there is still a meaningful point of human intervention, which is the laws of humanity. Designing and implementing those laws well remains an important challenge.

One last thought. I’ve been inspired by Beninger’s The Control Revolution (1986) which is a historical account of the information economy in terms of cybernetics and information theory. You can ask an AI to tell you more about it, but one item comes to mind: that each new information technology first seems to threaten the jobs of people doing information work, and then leads to an expanded number of information jobs. This has to do with the way complexity is and is not managed by the technology. There’s an open question whether this generation of AI is any different. The question is truly open, but my hunch at the moment is that today’s AI systems are creating a lot more complexity than they are controlling. We will see.

Leave a comment
January 2, 2026

naturalized ethics and natural law

One thing that’s become clear to me lately is that I now believe that ethics can be naturalized. I also believe that there is in fact a form of ‘natural law’. By this I mean that that there are rights and values that are inherent to human nature. Real legal systems can either lie up to natural law, or not.

This is not the only position that it’s possible to take on these topics.

One different position, that I do not have, is that ethics depends on the supernatural. I bring this up because religion is once again very politically salient in the United States. Abrahamic religions ground ethics and morality in a covenant between humans and a supernatural God. Divine power authorizes the ethical code. In some cases this is explicitly stated law, in others it is a set of principles. Beyond divine articulation, this position maintains that ethics are supernaturally enforced through reward and punishment. I don’t think this is how things work.

Another position I don’t have is that there is that ethics are opinion or cultural construction, full stop. Certainly there’s a wide diversity of opinions on ethics and cultural attitudes. Legal systems vary from place to place. This diversity is sometimes used as evidence that there aren’t truths about ethics or law to be had. But that is, taken alone, a silly argument. Lots of people and legal systems are simply wrong. Moreover, moral and ethical truths can take contingency and variety into account, and they probably should. It can be true that laws should be well-adapted to some otherwise arbitrary social expectations or material conditions. And so on.

There has historically been hemming and hawing about the fact/value dichotomy. If there’s no supernatural guarantor of ethics, is the natural world sufficient to produce values beyond our animal passions? This increasingly feels like an argument from a previous century. Adequate solutions to this problem have been offered by philosophers over time. They tend to involve some form of rational or reflective process, and aggregation over the needs and opinions of people in heterogeneous circumstances. Habermas comes to mind as a one of the synthesizers of a new definition of naturalized law and ethics.

For some reason, I’ve encountered so much resistance to this form of ethical or moral realism over the years. But looking back on it, I can’t recall a convincing argument for it. I can recall many claims that the idea of ethical and moral truth are somehow politically dangerous, but that this not the same thing.

There is something teleological about most viable definitions of naturalized ethics and natural law. They are would would hypothetically be decided on by interlocutors in an idealized but not yet realized circumstance. A corollary to my position is that ethical and moral facts exist, but many have not yet been discovered. A scientific process is needed to find them. This process is necessarily a social scientific process, since ethical and moral truths are truths about social systems and how they work.

It would be very fortunate, I think, if some academic department, discipline, or research institution were to take up my position. At present, we seem to have a few different political positions available to us in the United States:

  • A conservative rejection of the university of being insufficiently moral because of its abandonment of God
  • A postmodern rejection of ethical and moral truths that relativizes everything
  • A positivist rejection of normativity as the object of social science because of the fact/value dichotomy
  • Politicized disciplines that presume a political agenda and then perform research aligned with that agenda
  • Explicitly normative disciplines that are discursive and humanistic but not inclined towards rigorous analysis of the salient natural facts

None of these is conducive to a scientific study of what ethics and morals should be. There are exceptions, of course, and many brilliant people in many corners who make great contributions towards this goal. But they seem scattered at the margins of the various disciplines, rather than consolidated into a thriving body of intellect. At a moment where we see profound improvements (yes, improvements!) in our capacity for reasoning and scientific exploration, why hasn’t something like this emerged? It would be an improvement over the status quo.

4 Comments