Digifesto
July 26, 2012
The link between computation asymmetry and openness
I want to jot something down while it is on my mind. It’s rather speculative, but may wind up being the theme of my thesis work.
I’ve written here about computational asymmetry in the economy. The idea is that when different agents are endowed with different capacity to compute (or are differently boundedly rational)), then that can become an extreme inequality (power law distributed, as is income) as computational power is stockpiled as a kind of capital accumulation.
Whereas a solution to unequal income is redistribution and a solution to unequal physical is regulation against violence, for computational asymmetry there is a simpler solution: “openness” in the products of computation. In particular, high quality data goods–data that is computationally rich (has more logical depth)–can be made available as public goods.
There are several challenges to this idea. One is the problem of funding. How do you encourage the production of costly public goods? The classic answer is state funding. Today we have another viable option, crowdfunding.
Another involves questions of security and privacy. Can a policy of ‘openness’ lead to problematic invasions of privacy? Viewing the problem in light of computational assymetry sheds light into this dynamic. Privacy should be a privilege of the disempowered, openness a requirement of the powerful.
In an ideal economy, agents are rewarded for their contribution to social welfare. For high quality data goods, openness leads to the maximum social welfare. So in theory, agents should be willingly adopting an open policy of their own volition. What has prevented them in the past is transactions costs and the problem of incurred risk. As institutions that reduce transaction costs and absorb risks get better, the remaining problems will be ones of regulation of noncompetitive practices.
July 24, 2012
Why federally funded software should be open source
Recently, open access to government funded research has gained attention and traction. Britain and Europe have both announced that they will make research they fund open access. In the United States, a community-driven effort has pushed a Whitehouse petition to the Obama administration for a similar policy. We may be experiencing a sea change.
Perhaps on the coattails of this movement, Open Source for America has launched a petition asking for a similar policy regarded federally funded software development: share all government-developed software under an open source license.
This is a really good idea.
Unfortunately, software development and the government IT procurement are so misunderstood that this is not likely to excite those who aren’t somehow directly by the issue. That is too bad, because every American stands to benefit from this sort of change. That makes it important for those of us who do understand to act.
I’ll try to illustrate why this is important with a story, or really a template of a story. This is a story told in countless cases of government software procurement:
ACRNM, a federal agency, has realized that its database management system and its user interface have not been updated since the late 90’s, because building it the last time was such a headache. It never really worked the way they wanted it, and the vendor who built it for them has since vanished off the face of the earth. Desperate and beleaguered, ACRNM finally gets the budget together to build a new system, and put out a bid.
Vendors that have navigated the prerequisite bureaucratic maze flock to this bid, knowing victory will be lucrative. Among them is FUBAR Enterprise Solutions. They know that whatever they build, they have a revenue stream for life. Not only does ACRNM have an enormous internal incentive to declare the new system a success to justify their budget, but they also have nobody to turn to for help with their software when it inevitably fails but FUBAR. FUBAR can continue extorting ACRNM for cash until ACRNM gives up, and the cycle continues.
What is wrong with this picture? Let’s count the problems:
- FUBAR has ACRNM by the (pardon me, there’s really no other way to put this) balls. The term is vendor lock-in. The second ACRNM installs their system, FUBAR becomes a parasite on the government leeching taxpayer money. This is because the software is proprietary. No other company is legally allowed to fix or modify FUBAR’s proprietary system, so FUBAR faces no competition and so can charge through the nose. If the software were open source, ACRNM could turn to other contractors to repair their system, lowering total costs.
- ACRNM has to do its work with worse software. Remember, this is a government agency that we pay taxes to for their services. With so much government activity boiling down to bureaucratic information processing, and so much innovation in software engineering and design, and so much budgetary pressure, you would think that the federal government would leap at technological innovation. But proprietary contracting causes the government to cripple itself at a tipping point.
- Today, government agencies like ACRNM are wisening up and turning to open source solutions. But it’s a slow, slow process. This is partly because FUBAR and its buddy companies who, after so many years of this relationship with government, are now an entrenched lobby that will sow Fear, Uncertainty, and Doubt about open source alternatives if they can get away with it. In recent years, since open source has become more mainstream, these companies are admitting the viability of open source compatibility and mixed solutions. They see the writing on the wall. They will of course fight an open source purchasing mandate with everything they have.
- Few governmental problems are unique. If ACRNM is paying for a new custom software solution, there likely many other agencies–at federal, state, or local level–with a similar problem. Civic Commons has already jumped on this opportunity by trying to facilitate technology reuse across city governments. If ACRNM invests in an open source solution, then other agencies can seek out that solution and adapt it to their needs, reducing government IT costs overall.
- As we’ve discussed, open source software creates a competitive market for services. That makes an open source mandate a job creation program. Every new open technology is an opportunity for several small businesses to open. These are businesses that share fixed costs to market entry and add value through technologist consulting and custom development. Jobs customizing existing open source solutions can be well-paid with even an entry-level programming skill set, and are a good way to build a lasting career in the technology sector. Federal investment in open source software builds our national supply of technology skill faster than proprietary investment.
- Lastly, but certainly not least, is the possible reuse of open source technology by the private sector. Just as federally funded research contributes to growth in America’s scientific industry, federal investment in software provides a foundation for stronger tech companies. Openness in both cases expands the impact of the funding.
So, to recap: if this sort of policy passes, the winners are government employees, taxpayers, entry-level workers with a minimum of technical skills, and the tech industry in general. The losers (in the short term) are those existing companies that have the federal government locked into custom proprietary software contracts.
I want to make a point clear: I am talking specifically about new software development in this post. Purchasing licenses for existing proprietary software is a different story.
Brian Carver, professor at UC Berkeley School of Information, has offered this clarification of what an open source mandate could look like:
- An unambiguous policy and awareness that all software created by
federal employees as part of their job duties is not subject to copyright
at all and is born in the public domain, and therefore not subject to any
license terms at all, including a FOSS license.- Given 1, the federal government should either just use github/bitbucket
or set up a similar repository to share all such federal government
software that is in the public domain.- When the federal government contracts with developers for software,
there should be an unambiguous policy that all such software must be
licensed under a FOSS license unless subject to a specifically-requested
exemption (national security, military, etc.)
A central election issue is the size and role of government in the economy. Politicians on the right advocate for smaller government and a strong private sector with competitive markets. Politicians on the left advocate for government’s active investment in the economy.
Proprietary government-developed software is the worst of both worlds: inefficient government spending to create parasitic, uncompetitive companies that don’t invest their technology back into the economy. An open source mandate would give us the best of both worlds: efficient government spending that shrinks government (by easing overhead) while investing in new technology and competitive businesses.
The movement for open access to government funded research is strong and winning victories around the world. Maybe we can do the same for government funded software development.
July 22, 2012
The Shame or Shine Lotto
Consider the following Massively Multiplier On-line Game:
- The game is strictly opt in. Nobody is forced to play the game.
- Upon joining, some set of personal details is tracked and saved by the game. Purchasing data, tax records, …hell, legal record, personal messages?
- Once per day, N players are selected at random and the data available on them are released into the public domain.
- Members can look up to see whether others are playing the game. In addition to identifying information, they can see what information a player has agreed to have tracked.
It’s the Shame or Shine Lotto! Every day, there is a chance you will be roasted or toasted for the information you’ve agreed to uncertainly share.
Would you play this game?
July 9, 2012
We need help naming a software project
Speaking of computational argumentation, Dave Kush and I are starting a software project and we need a name for it.
The purpose of the software is to extract information from a large number of documents, and then merge this information together into a knowledge base. We think this could be pretty great because it would support:
- Conflict detection and resolution. In the process of combining information from many sources into a single knowledge base, the system should be able to mark conflicts of information. That would indicate an inconsistency or controversy between the documents, which could be flagged for further investigation.
- Naturally queryable aggregate knowledge. We anticipate being able to build a query interface that is a natural extension of this system: just run the query through the extraction process and compare the result for consistency with the knowledge base. This would make the system into a “dissonance engine,” useful for opposition research or the popping of filter bubbles.
I should say that neither of us knows exactly what we are doing. But Dave’s almost got his PhD in human syntax so I think we’ve got a shot at building a sweet parser. What’s more, we’ve got the will and plan. It will be open source, of course, and we’re eager for collaborators.
We have one problem:
We don’t know what to call it.
I can’t even make the GitHub account for our code until we have a good name. And until then we’ll be sending Python scripts to each other as email attachments and that will never get anywhere.
Please help us. Tell us what to name our project. If we use your name, we’ll do something awesome for you some day.
Scratch that. We’re calling it Bluestocking. The GitHub repo is here.
July 8, 2012
Computational Asymmetry
I’ve written a paper with John Chuang about “Computational Asymmetry in Strategic Bayes Networks” to open a conversation about an economic and social issue: computational asymmetry. By this I mean the problem that some agents–people, corporations, nations–have access to more computational power than others.
We know that computational power is a scarce resource. Computing costs money, whether we buy our own hardware or rent it on the cloud. Should we be concerned with how this resource gets distributed in society?
One could argue that the market will lead to an efficient distribution of computing power, just like it leads to an efficient distribution of brown shoes or butter. But that argument only makes sense if computational power is not associated with externalities that would cause systematic market failure.
This isn’t likely. We know that information asymmetry can wreak havoc on market efficiency. Arguably, computational asymmetry is another form of information asymmetry: it allows some parties to get important information, faster. Or perhaps a better way to put it is that with more computing power, you can get more knowledge out of the information you already have.
In the paper linked above, we show that in some game theoretic situations with complex problems, more computationally powerful players can beat their opponents using only their superior silicon. Suppose that organizations use computing power to gain an economic advantage, and then use their winnings to invest in more computing power? You could see how this cycle would lead to massive inequality.
I don’t think this situation is far fetched. In fact, we may already be living it. Consider that computing power is carried not just by hardware availability, but by software and human capital. What are the most powerful forces in United States politics today? Is it Wall Street, with its bright minds and high-frequency traders? Or Silicon Valley, crunching data and rolling out code? Or technocratic elites in government? President Obama has a large team of software developers available to build whatever data mining tools he needs. Does Mexico have the same skills and tools at its disposal? Does Nigeria? There is asymmetry here. How will this power imbalance manifest itself in twenty years? Fifty years?
Henry Farrell (George Washington University) and Cosma Rohilla Shalizi (Carnegie-Mellon/The Santa Fe Institute) have recently put out a great paper about Cognitive Democracy, a political theory that grapple’s with society’s ability to solve complex problems. Following Hayek, who maintains that the market will efficiently solve complex economic problems, and Thaler and Sunstein, who believe that a paternalistic hierarchy can solve problems in a disinterested way, Farrell and Shaliza argue that a radical democracy can solve problems in a way that diffuses unequal power through people’s confrontation with other viewpoints. This requires that open argumentation and deliberation being an effective information-processing mechanism. They advocate for greater experimentation with democratic structure over the Internet, with the goal of eventually re-designing democratic institutions.
I love the concept of cognitive democracy and their approach. However, if their background assumptions are correct then computational asymmetry poses a problem. Politics is the negotiation of adversarial interests. If argumentation is a computational process (which I believe it is), then even a system of governance based on free speech and collective intelligence could be manipulated or overpowered by a computational titan. In such a system, whoever holds the greatest gigahertz gets a bigger piece of the derived social truth. As we plunge into a more computationally directed world, that should give us pause.
April 11, 2012
Defining information with Dretske
I prepared these slides to present Fred Dretske’s paper “The Epistemology of Belief” to a class I’m taking this semester, ‘Concepts of Information’, taught by Paul Duguid and Geoff Nunberg.
Somewhere along the line I realised that if I was put on earth for one reason and one reason only, it was to make slide decks about epistemology.
I’ve had a serious interest in philosophy as a student and as a…hobbyist? can you say that?…for my entire thinking life. I considered going to graduate school for it before tossing the idea for more practical pursuits. So it comes as a delightful surprise that I’ve found an opportunity to read and work with philosophy at a graduate level through my program.
A difficult issue for a “School of Information” is defining what information is. I’ve gathered from conversations with faculty that there is an acknowledged intellectual tussle over the identity of iSchools which hinges in part on the meaning of the word. There seems to me to be roughly two ideologies at play: the cyberneticist ideology that sought to unify Shannon’s information theory, computer science, management science, economics, AI, and psychology under a coherent definition of information on the one hand, and the softer social science view that ‘information’ is a polysemous term which refers variously to newspapers and the stuff mediated by “information technology” in a loose sense but primarily as a social phenomenon.
As I’ve been steeped in the cyberneticist tradition but still consider myself literate in English and capable of recognizing social phenomena, it bothers me that people don’t see all this as just talking about the same thing in different ways.
I figured coming into the program that this was an obvious point that was widely accepted. It’s in a way nice to see that this is controversial and the arguments for this view are either unknown, unarticulated, or obscure, because that means I have some interesting work ahead of me.
This slide deck was a first stab at the problem: tying Dretske’s persuasive account of a qualitative definition of ‘information about’ to the relevant concept of Shannon’s information theory. I hope to see how far I can push this in later work. (At the point where is proves impossible, as opposed to merely difficult or non-obvious, then we’ll have discovered something new!)
March 8, 2012
Academia vs. FOSS: The Good, The Bad, and the Ugly
Mel Chua has been pushing forward on the theme of FOSS culture in academia, and has gotten a lot of wonderful comments, many about why it’s not so simple to just port one culture over to the other. I want to try to compile items from Mel, comments on that post, and a few other sources. The question is: what are the salient differences between FOSS and academia?
I will proceed using the now-standard Spaghetti Western classification schema.
The Good
- Universities tend to be more proactive about identifying and aiding newcomers that are struggling, as opposed to many FOSS projects that have high failure-and-dropout rates due to poorly designed scaffolding.
- Academia is much more demographically inclusive. FOSS communities are notoriously imbalanced in terms of gender and race.
The Bad
- The academic fear of having ones results scooped or stolen results in redundant, secrecy, and lonely effort. FOSS communities get around this by having good systems for attribution of incremental progress.
- Despite scientific ideals, academic scientific research is getting less reproducible, and therefore less robust, because of closed code and data. FOSS work is often more reproducible (though not if its poorly documented).
- Closed access academic journals hold many disciplines hostage by holding a monopoly on prestige. This is changing with the push for open access research, but this is still a significant issue. FOSS communities may care about community prestige, but often that prestige comes from community helpfulness or stake in a project. If metrics are used, they are often implicit ones extractable from the code repository itself, like Ohloh. Altmetrics are a solution to this problem.
The Ugly
- In both FOSS and academia, a community of collaborators needs to form around shared interests and skills. But FOSS has come to exemplify the power of the distributed collaboration towards pragmatic goals. One is judged more by ones contributions than by ones academic pedigree, which means that FOSS does not have as much institutional gatekeeping.
- Tenure committees look at papers published, not software developed. So there is little incentive for making robust software as part of the research process, however much that might allow reproducibility and encourage collaboration.
- Since academics are often focused on “the frontier”, they don’t pay much attention to “building blocks”. Academic research culture tends to encourage this because it’s a race for discovery. FOSS regards care of the building blocks as a virtue and rewards the effort with stronger communities built on top of those blocks.
- One reason for the difference between academia and FOSS is bandwidth. Since publications have page limits and are also the main means of academic communication, one wants to dedicate as much space as possible to juicy results at the expense of process documentation that would aid reproducibility. Since FOSS developed using digital communication tools with fewer constraints, it doesn’t have this problem. But academia doesn’t yet value contributions to this amorphous digital wealth of knowledge.
Have I left anything out?
March 7, 2012
Don’t use Venn diagrams like this
Today I saw this whitepaper by Esri about their use of open source software. It’s old, but still kept my attention.
There’s several reasons why this paper is interesting. One reason is that it reflects the trend of companies that once used FUD tactics around open source software to singing a soothing song of compatibilism. It makes an admirable effort to explain the differences between open source, proprietary software, and open standards to its enterprise client audience. That is the good news.
The bad news is that since this new compatibilism is just bending to market pressure after the rise of successful open source software complements, it lacks an understanding of why the open source development process has caused those market successes. Of course, proprietary companies have good reason to blur these lines, because otherwise they would need to acknowledge the existence of open source substitutes. In Esri’s case, that would mean products like the OpenGeo Suite.
I probably wouldn’t have written this post if it were not for this Venn diagram, which is presented with the caption A hybrid relationship:
I don’t think there is a way to interpret this diagram in a way that makes sense. It correctly identifies that Closed Source, Open Source, and Open Standards are different. But what do the overlapping regions represent? Presumabely they are meant to indicate that a system may both be open source and use open standards, or have open standards and be closed, or…be both open and closed?
It’s a subtle point but the semantics of set containment implied by the Venn diagram really don’t apply here. A system that’s a ‘hybrid’ between a closed and open software is not “both” closed and open the same way closed software that uses open standards is “both” closed and open. Rather, the hybrid system is just that, a hybrid, which means that its architecture is going to suffer tradeoffs as different components have different properties.
I don’t think that the author of this whitepaper was trying to deliberately obscure this idea. But I think that they didn’t know or care about it. That’s a problem, because it’s marketing material like this that clouds the picture about the value of open source. At a pointy-haired managerial level, one can answer the question “why aren’t you using more open source software” with a glib, “oh, we’re using a hybrid model, tailored to our needs.” But unless you actually understand what you’re talking about, your technical stack may still be full of buggy and unaccountable software, without you even knowing it.
