Digifesto

Tag: AI

July 22, 2026

The Open Silicon Fallacy: How Panic Over Chinese Open-Weight Models Is Distorting American AI Policy

The following article was written with Gemini Flash 3.6 “in the style of The Economist’s Bagehot”. It is an experiment in AI writing. The arguments and structure are mine. – S

I. The Dragons in the Weights

In the summer of 2026, the fashionable anxiety in Washington and Silicon Valley is that Western technological supremacy has been undermined by a collection of downloadable matrix files. When Chinese research labs like DeepSeek, Moonshot AI, and Z.ai released the weights for their latest systems, the initial response from American technology executives was a nervous cough about benchmark scores. When developers discovered these open-weight models could execute multi-step reasoning at a fraction of the cost of renting access to proprietary American cloud endpoints, the nervous cough turned into a geopolitical emergency.

The narrative is simple, compelling, and decidedly panicked. Chinese open-weight architectures are closing the benchmark gap with closed Western labs. Engineering teams weary of driving a metaphorical Ferrari to Whole Foods for basic data transformations are migrating toward open models hosted locally or behind private firewalls.

To visit Capitol Hill today is to encounter three distinct pillars of alarm regarding these releases. First comes the fear of lost technological primacy, as the arrival of competitive Chinese models shatters the illusion that GPU export restrictions would maintain a multi-generational lead. Second comes the specter of standards hegemony, with strategists dreading a world where global software standardizes on Chinese-developed open architectures. Third, and most loudly invoked, comes the proliferation of dual-use capabilities, as downloadable intelligence primitives escape centralized censorship and safety filters.

Yet what is being distributed from Beijing is not open source in the classic sense of transparent codebases and reproducible pipelines. These are open weights—the pre-computed mathematical parameters of deep neural networks whose underlying data and alignment routines remain closely guarded secrets. That they are nevertheless transforming global software architecture says far more about the economics of general computing than about the ideological triumph of open-source idealism.

II. Bootleggers, Baptists, and Moats

It is a sound rule of political economy that whenever a commercial sector and a national security agency begin using identical language to describe a threat, one should look closely at who stands to profit from the proposed remedy.

What Washington is currently witnessing is a classic demonstration of the bootleggers and baptists dynamic. The baptists are the national security hawks, genuinely concerned with technological primacy and cyber-resilience. The bootleggers are the dominant proprietary API vendors, whose staggering valuations depend entirely on convincing the market that intelligence can only be safely consumed as a paid subscription utility.

When proprietary labs lobby for mandatory model licensing, pre-deployment government safety audits, or outright bans on foreign open-weight distributions, they do so under the pious banner of national defense. Yet the operational effect of such proposals is unmistakable, erecting regulatory moats that entrench an oligopolistic duopoly. By framing open-weight distribution as an inherent national security threat, incumbents seek to achieve through regulatory capture what they struggle to maintain through market competition: the enclosure of the AI stack. The public interest is conflated with the profit margins of cloud providers, while the broader software ecosystem is instructed to accept API lock-in as a patriotic duty.

III. The Halloween Documents Revisited

History does not repeat itself, but software executives certainly recycle their memo templates. In the late 1990s, when Microsoft felt its desktop monopoly threatened by Linux, its executives authored internal strategic assessments—the famous Halloween Documents—warning that open software presented a systemic threat to software stability, intellectual property, and commercial viability.

The current rhetoric against open-weight AI models reproduces this playbook line for line. Once again, open distribution is framed as an irresponsible hazard; once again, security through obscurity is held up as the only responsible posture.

Yet incumbent resistance follows a predictable lifecycle, beginning with initial ridicule, progressing to intense alarmism, moving to lobbying for legal restriction, and ultimately settling into a quiet, pragmatic pivot to co-optation. Two decades after penning memos declaring open software an existential cancer, Microsoft spent 7.5 billion dollars to acquire GitHub, transforming itself into the world’s largest host of open-source code. The very proprietary companies that once swore open software was a menace today run their cloud empires on open infrastructure.

IV. How the Defense State Learned to Love the Kernel

The irony of the current policy panic is that the national security establishment has already solved this problem once before. In the early days of networked computing, defense agencies viewed open-source software with profound suspicion, assuming closed, proprietary systems were superior because their source code was hidden behind non-disclosure agreements and commercial firewalls.

By the early 2000s, however, military and intelligence strategists realized that proprietary vendors could not patch vulnerabilities or adapt to new threats as rapidly as a global community of developers inspecting open code. The intelligence community embraced the doctrine of security through visibility. In 2000, the National Security Agency took an open-source Linux kernel, added mandatory access controls directly into its architecture, and handed Security-Enhanced Linux back to the public.

National security interests accommodated open source not by suppressing it, but by co-opting, hardening, and building on top of it. They recognized that controlling an open, auditable standard offered greater agility and defense-in-depth than relying on a commercial black box.

V. Compilers, Not Missiles

The current attempt to govern AI safety by restricting model weights rests on a fundamental misapprehension of what a large language model actually is. Policy makers consistently treat probabilistic language models as if they were self-contained, autonomous products or guided weapons systems that can be aligned at the factory and locked in a box. In reality, a foundation model is a general-purpose computing primitive, serving as the statistical equivalent of a C compiler or an arithmetic logic unit for natural language and code.

Trying to enforce safety at the weight level is as ineffective as trying to secure an operating system by banning specific sequences of assembly language instructions. Raw inference is inherently difficult to control at the parameter level; a model that can write a Python script for a database query can, with minimal prompting, write a script to probe a network port.

Opponents of open weights often argue that publicly downloadable parameters allow offline execution, rendering traditional hardware tracking obsolete. This argument, however, confuses hardware tracking with runtime deployment governance. The true execution boundary is not the local matrix multiplication happening on a graphics card, but the point where an AI system interacts with the real world through API keys, database access, tool-use privileges, and execution environments. Smart policy does not attempt to police raw matrix math in memory; it enforces strict sandboxing, zero-trust permissions, and identity verification at the application layer where actions occur.

VI. Critiquing “Openness”

To defend the availability of open weights is not to be naive about their limitations. Indeed, neural network weights occupy a strange conceptual middle ground. They are not traditional open source code, nor are they merely untrusted software binaries; they are dense, highly compressed mathematical encodings of vast cultural, technical, and linguistic corpora. They cannot be read in any conventional human sense like C instructions, yet they contain whole libraries of distilled human data.

This epistemic opacity is precisely why open weights are necessary. Having direct access to raw model parameters is the strict prerequisite for mechanistic interpretability, local safety probing, and post-hoc red-teaming. A proprietary API offers zero visibility into what lies behind the endpoint, whereas an open weight file allows security researchers to inspect internal activation patterns, trace knowledge representations, and strip out toxic behaviors.

Similarly, open-weight models originating from authoritarian states are indisputably shaped by domestic censorship and potential state alignment. The correct operational response, however, is to treat them with the same caution accorded to foreign-built infrastructure. Western developers can download, audit, strip out state-imposed guardrails, and repurpose foreign base parameters for independent domestic use, turning foreign releases to Western defensive advantage.

As scholars David Gray Widder, Meredith Whittaker, and Sarah Myers West point out in Nature (2024), tech giants frequently engage in openwashing—releasing model weights as a public relations gesture while keeping training datasets, filtering pipelines, and compute infrastructure firmly closed. This critique is vital, but its policy conclusion must be drawn carefully. That open-weight releases represent an incomplete form of openness is an argument for demanding greater transparency and public investment in shared compute and datasets. It is emphatically not an argument for retreating into the arms of proprietary API monopolies.

VII. The Open Security Imperative

The present impulse to restrict, license, or ban open-weight AI models repeats the classical errors of past technological panics. It mistakes corporate rent-seeking for national defense, confuses general computing primitives with finished weapons, and trades long-term systemic resilience for the illusion of central control.

A pragmatic blueprint for AI policy must start from a posture of realism. General-purpose reasoning parameters will circulate globally across open networks regardless of administrative bans. The defense of critical infrastructure relies on open access to model parameters, enabling global researchers to discover vulnerabilities and build defensive countermeasures faster than adversaries can exploit them. Policy must focus its regulatory instruments on the environment where software acts—governing identity verification, agentic tool permissions, data access, and sandboxed execution layers—rather than attempting to criminalize the distribution of general-purpose math.

To lock down American AI within proprietary walled gardens out of fear of foreign open-weight competition would be a historic miscalculation. In the long struggle for technological adaptability and national security, open systems remain, as they have always been, the ultimate line of defense.

References

Leave a comment
June 13, 2026

LLMs as computation

LLMs are now”doing” a lot of technical system design and are the object of a great deal of computer science research. However, I’ve surprised by much of the research that crosses my way (admittedly likely not a great sample) treats LLMs as a general form of intelligence without treating it as a form of computation. I expect that some combination of theory of computation (such as algorithmic information theory) and structural economics is needed to get a rigorous handle on the AI economy. This blog post contains some notes toward this end.

As we all know, an LLM is a collection of neural network weights, trained on a massive amount of information, which consumes tokens and emits predicted next tokens. Simplifying a bit, we can model an LLM as a machine that, given a string of tokens, emits a string of tokens.

Let Σ\Sigma be the set of tokens, Σ∗\Sigma^* be the space of token strings of any length. Perhaps an LLM is a function:

L:Σ∗→Σ∗L: \Sigma^* \rightarrow \Sigma^*

Really, this is LLM “inference”. I’m omitting the inherent stochasticity of LLMs — more realistically, LL would be a conditional probability distribution. But leave that aside for now.

Assuming that LL can consume as input any string, and in principle produce as output any string, what we have here is a class of “universal programming language”, another formal mathematical construct. “universal programming languages” appear in algorithmic information theory.

The simplest form of “universal programming language” is the print function. It repeats as output anything put into it. People (including myself) once joked that LLMs are glorified autocomplete; they clearly do more than this. The weights must matter.

Really, LLMs are parameterized functions — the parameters θ\theta are weights of the neural network.

Lθ:Σ∗→Σ∗L_\theta: \Sigma^* \rightarrow \Sigma^*

The weights are a compression of a great deal of training data 𝐃\mathbf{D}. Let’s assume training has converted this data to a set of weights T(𝐃)→𝛉T(\mathbf{D}) \rightarrow \mathbf{\theta}. We can refer to this foundation model as 𝐋𝛉\mathbf{L_\theta} or 𝐋𝐃\mathbf{L_D}.

What else can you do with these models? You can provide them ‘context’ — additional strings as input. You can fine-tune them on more data. And you can use them for ‘reasoning’ by chaining inputs and outputs.

  • Context: allow multiple string inputs Lθ(c,i)→oL_\theta(c, i) \rightarrow o
  • Fine-tuning: T(LD,d)→LD+dT(L_D, d) \rightarrow L_{D + d} — further compresses additional data dd into the model weights
  • Reasoning: Lθn(i)→Lθ(Lθ(...(Lθ(i)))→oL^n_\theta(i) \rightarrow L_\theta(L_\theta(…(L_\theta(i))) \rightarrow o applies the model recursively nn times

So if we want to look at the data and computation pipeline of an LLM based system, we get something like:

(Tn(D,d1,...dn))m(c,i)→o(T^n(D,d_1, …d_n))^m(c,i) \rightarrow o

I.e., we train on a base data set and several fine-tuning data sets, pick context and an input, and run inference some number of times. Each of these steps has a cost function, and we can then computer the average costs of solving various sets of problems given the available data, and other statistics. This then can be used to design the most efficient pipelines and markets.

I would be interested in hearing from anybody about whether and how this faithfully captures the essentials of LLMs as a form of computation. This is my ‘mental model’. I have left out tool use and interactivity, among other things, but those can be added in easily.

Why am I writing this? Because I think that clearly articulating the formal properties of LLMs brings a number of issues to light.

First, it foregrounds the importance of training data. Famously, the transformer architecture is very general, and early innovation in LLMs was largely about scaling it up to greater amounts of data. If we are interested in the behavior of LLMs, the training data and the training algorithm are the parts that are not “black boxes” to the model creators.

As we look at the future of LLMs in the economy, we will be looking at the results of differential access to data, as well as what data is commonly available. This shares a lot of patterns with previous iterations of concerns over “big data”, but this is obscured today because of the charisma of the models themselves.

Second, it makes explicit how information can flow and transform into a system output. The information comes first from training and fine-tuning data, then from context, then from system input. If the training and inference algorithms are general enough, none of the information relevant to a specific task comes from those parts of the system. Those algorithms are ‘general computing’.

Third, it breaks up training and inference. While training and inference are not so different in terms of information flow, they are in practice quite different because of their physical and economic costs. Currently, training is more expensive than inference. So, we see a race to, expensively, train general models with more and more data, so that less and less data is needed in context at inference time, and fewer steps are needed during reasoning. A structural model that distinguishes these can discriminate between several investment hypotheses in this space.

Fourth, by revealing LLMs as a form of general data processing and computation, it deflates (in what I think is a good and necessary way) the tendency to see ‘model evaluations’ as the best way to enforce AI accuracy, fairness, privacy, and so on. My general frustration with the model evaluation literature is that LLMs are that if they are a flavor of universal programming language by design, then there will, by definition, always be a jailbreak or a hallucination available to them. A lot of work on ‘guardrails’ at the model level seems to be about making certain kinds of outputs more difficult or expensive to get. As we’ve seen, there will be open models, and they will get fine-tuned by hobbyists and others to get around the guardrails, and so that’s not going to be an effective strategy long term.

This means that a lot of AI product design and regulation seems to be about shifting around the cost functions for achieving certain kinds of outputs with certain data. If ‘bad’ behaviors are expensive, and ‘good’ behaviors are cheap, then we have, in a sense, succeeded. But this means that the underlying economics must be part of the analysis for it to have forward-going relevance and replicability. Today’s model capabilities are a function of whatever the latest investment — at the training and inference level, as well as the data flow of context and inputs, which may go back into training — is. The entire pipeline produces ‘the intelligence’, and it does so at physical and economic cost. Computer science research, per se, with its focus on the currently available digital artifacts, is not going to achieve lasting results unless it expands its purview to these broader systems and considerations. Likewise, evaluations of models alone will not provide us the reliable theoretical knowledge needed to steer public policy. We must take into account production costs and data pipelines.

Leave a comment
February 17, 2026

updates and stubbornness about superintelligence

We seem to be in a new moment of media excitement about the implications of artificial intelligence. This time, the moment is driven by the experience of software engineers and other knowledge workers who are automating their work with ‘agents’. Clause Code etc. The latest generation of models and services is really good at doing things.

Does this change anything about my “position on AI” and superintelligence in particular?

I wrote a brief paper in 2017 about Bostrom’s Superintelligence argument. I concluded that algorithmic self-improvement at the software level would not produce superintelligence. Rather, intelligence group is limited by data and hardware.

In 2025, this conclusion still holds up, as we’ve seen that the recent impressive advances in AI has depended on tremendous capital expenditure on data centers, high-performing chips, and energy. It also depends on well-publicized efforts to collect all the text known to humankind for training data.

About 8 years ago when I was thinking about this, I wrote a bit about the connection between the Superintelligence argument and the Frankfurt School’s views on instrumental reason and capitalism. The alignment of AI with capital has born out, and has been written about by many others. What is striking about the current moment is just how on-the-nose that alignment is in the US, in terms of the full stack of energy, hardware, models, applications, and then some.

So, so far, no update.

In 2021 I published an article saying that we already had artificial systems with the capacity to outperform individual humans at many tasks. They were and still are called corporations or firms. We also had replaced markets with platform, which are similarly more performant in terms of reducing transaction costs. In that article, Jake Goldenfein and I argue that what ultimately matters are the purposes of the social system that operates the AI technology.

I believe this argument also continues to hold up. The successful models and service we are seeing are corporate accomplishments. The corporation is still the relevant unit of analysis when considering AI.

There are a number of interesting things happening now which I think are undertheorized:

  • What is the real economics of AI, given that the supply chains are so long and complex, consistent of both material and intellectual inputs, and the market for demand is uncertain? This is the trillion dollar question in terms of valuations, and it’s unanswered. The empirics here are not very good because things are far out of equlibrium.
  • Put another way: what does AI mean for the relationships between capital, corporations, labor, and consumers? Some of these relationships are mediated by rules about corporate law, intellectual property and data use, and so are determinable by law rather than technology. Information law therefore is a key point of political intervention in an economic system that is otherwise determined by laws of nature (energy, computation, etc.?

To put it another way: superintelligence has been happening and continues to happen. Some of this is due to laws of nature. But there is still a meaningful point of human intervention, which is the laws of humanity. Designing and implementing those laws well remains an important challenge.

One last thought. I’ve been inspired by Beninger’s The Control Revolution (1986) which is a historical account of the information economy in terms of cybernetics and information theory. You can ask an AI to tell you more about it, but one item comes to mind: that each new information technology first seems to threaten the jobs of people doing information work, and then leads to an expanded number of information jobs. This has to do with the way complexity is and is not managed by the technology. There’s an open question whether this generation of AI is any different. The question is truly open, but my hunch at the moment is that today’s AI systems are creating a lot more complexity than they are controlling. We will see.

Leave a comment
October 3, 2024

I miss writing

I miss writing.

For most of my formative years, writing was an important activity in my life. I would try to articulate what I cared about, the questions I had, what I was excited for. Often this was done as an intellectual pursuit, something para-academic, a search for answers beyond any curriculum, and without a clear goal in mind.

I wrote both in private and “in public”. Privately, in paper journals and letters to friends, then emails. “In public”, on social media of various forms. LiveJournal was a fantastic host for pseudonymous writing, and writing together and intimately with others was a shared pastime. These were the ‘old internet’ days, when the web was imagined as a frontier and creative space.

The character of writing changed as the Web, and its users, matured. Facebook expanded from its original base of students to include, well, everybody. Twitter had its “Weird Twitter” moment, which then passed as it became an increasingly transactional platform. Every public on-line space became a LinkedIn.

Instagram is, for a writer, depressing. Extremely popular and influential, but with hardly any written ideas, by design. YouTube can accommodate writing, but only with the added production value of performance, recording, and so on. It’s a thicker medium that makes writing, per se, seem small, or shallow.

The Web made it clear that all text is also data. Writing is a human act, full of meaning, but digital text, the visible trace of that act, is numerical. Text that is published online is at best the body of a message with so much other metadata. The provenance is, when the system is working, explicit. The message is both intrinsically numerical — encoded in bits of information — and extrinsically numerical: its impact, reach, engagement, is scored; it is indexed and served as a result to queries based on its triangulated position in the vast space of all possible text.

Every day we are inundated with content, and within the silicon cage of our historical moment, people labor to produce new content, to elevate their place in the system of ranks and numbers. I remember, long ago, I would write for an imagined audience. Sometimes this audience was only myself. This was writing. I think writing is rarer now. Many people create content and the audience for that content is the system of ranks and numbers. I miss writing.

Generative Artificial Intelligence is another phase of this evolution. Text is encoded, intrinsically, into bits. Content is sorted, extrinsically, into tables of impact, engagement, conversion, relevance, and so on. But there had been a mystery, still, about the way words were put together, and what they meant when they came in this or that order.

That mystery has been solved, they say. See, my looking at all the text at once, each word or phrase can be seen as a vector in a high dimensional space, relative to all other texts. The meaning of the text is where it is in that space, as understood by a silent, mechanical observer of all texts available. Relative to a large language model, the language in one person’s mind is small, shallow.

Generative AI now excels at creating content. The silicon cage may soon start evicting its prisoners. Their labor isn’t needed any more. The numbers can take care of themselves.

I remember how to write. I now realize that what is special about writing is not that it produces text — this is now easily done by machines. It is that it is a human act that transforms the human as writer.

I wonder for how long humanity will remember how to read.

Leave a comment
September 28, 2012

dreams of reason

Begin insomniac academic blogging:

Dave Lester has explained his strategy in graduate school as “living agily”, a reference to agile software development.

In trying to navigate the academic world, I find myself sniffing the air through conversations, email exchanges, tweets. Since this feels like part of my full time job, I have been approaching the task with gusto and believe I am learning rapidly.

Intellectual fashions shift quickly. A year ago I first heard the term “digital humanities”. At the time, it appeared to be controversial but on the rise. Now, it seems like something people are either disillusioned with or pissed about. (What’s this based on? A couple conversations this week, a few tweets. Is that sufficient grounds to reify a ‘trend’?)

I have no dog in that race yet. I can’t claim to understand what “digital humanities” means. But from what I gather, it represents a serious attempt to approach text in its quantitative/qualitative duality.

It seems that such a research program would: (a) fall short of traditional humanities methods at first, due to the primitive nature of the tools available, (b) become more insightful as the tools develop, and so (c) be both disgusting and threatening to humanities scholars who would prefer that their industry not be disrupted.

I was reminded through an exchange with some Facebook Marxists that Hegel wrote about the relationship between the quantitative and the qualitative. I forget if quantity was a moment in transition to quality, or the other way around, or if they bear some mutual relationship, for Hegel.

I’m both exhausted about and excited that in order to understand the evolution of the environment I’m in, and make strategic choices about how to apply myself, I have to (re?)read some Hegel. I believe the relevant sections are this and this from his Science of Logic.

This just in! Information about why people are outraged by digital humanities!

There we have it. Confirmation that outrage at digital humanities is against the funding of research based on the assumption that “that formal characteristics of a text may also be of importance in calling a fictional text literary or non-literary, and good or bad”–i.e., that some aspects of literary quality may be reducible to quantitative properties of the text.

A lot of progress has been made in psychology by assuming that psychological properties–manifestly qualitative–supervene on quantitatively articulated properties of physical reality. The study of neurocomputation, for example, depends on this. This leads to all sorts of cool new technology, like prosthetic limbs and hearing aids and combat drones controlled by dreaming children (potentially).

So, is it safe to say that if you’re against digital humanities, you are against the unremitting march of technical progress? I suppose I could see why one would be, but I think that’s something we have to take a gamble on, steering it as we go.

In related news, I am getting a lot out of my course on statistical learning theory. Looking up something I wanted to include in this post just now about what I’ve been learning, I found this funny picture:

One thing that’s great about this picture is how it makes explicit how, in a model of the mind adopted by statistical cognitive science theorists, The World is understood by us through a mentally internal Estimator whose parameters are strictly speaking quantitative. They are quantitative because they are posited to instantiate certain algorithms, such as those derived by statistical learning theorists. These algorithmic functions presumably supervene on a neurocomputational substrate.

But that’s a digression. What I wanted to say is how exciting belief propagation algorithms for computing marginal probabilities on probabilistic graphical models are!

What’s exciting about them is the promise they hold for the convergence of opinion onto correct belief based on a simple algorithm. Each node in a network of variables listens to all of its neighbors. Occasionally (on a schedule whose parameters are free for optimization to context) the node will synthesize the state of all of its neighbors except one, then push that “message” to its neighbor, who is listening…

…and so on, recursively. This algorithm has nice mathematically guaranteed convergence properties when the underlying graph has no cycles. Meaning, the algorithm finds the truth about the marginal probabilities of the nodes in a guaranteed amount of time.

It also has some nice empirically determined properties when the underlying graph has cycles.

The metaphor is loose, at this point. If I could dream my thesis into being at this moment, it would be a theoretical reduction of discourse on the internet (as a special case of discourse in general) to belief propagation on probabilistic graphical models. Ideally, it would have to account for adversarial agents within the system (i.e. it would have to be analyzed for its security properties), and support design recommendations for technology that catalyzed the process.

I think it’s possible. Not done alone, of course, but what projects are ever really undertaken alone?

Would it be good for the world? I’m not sure. Maybe if done right.

2 Comments