Tag: reproducibility

instrumental realism and reproducibility in science and society

In Instrumental Realism, Ihde does a complimentary treatment of Ackerman’s Data, Instruments, and Theory (1985), which is positioned as a rebuttal to Kuhn. It is a defense of the idea of scientific progress, which is so disliked by critical scholarship. The key issue is are relativistic attacks on scientific progression that point out, for example, the ways in which theory shapes observation, which undermines the objectivity of observation. Ackerman’s rebuttal is that science does not progress through advance of theory, but rather through advance of instrumentation. Instruments allow data to be collected independently of theory. This creates and bounds “data domains”–fields of “data text” that can then be the site of scientific controversy and resolution.

The paradigmatic scientific instruments in Ackerman’s analysis are the telescope and the microscope. But it’s worthwhile thinking about what this means for the computational tools of “data science”.

Certainly, there has been a great amount of work done on the design and standardization of computational tools, and these tools work with ever increasing speed and robustness.

One of the most controversial points made in research today is the idea that the design and/or of these computational tools encodes some kind of bias that threatens the objectivity of their results.

One story, perhaps a straw man, for how this can happen is this: the creators of these tools have (perhaps unconscious) theoretical presuppositions that are the psychological encoding of political power dynamics. These psychological biases impact their judgment as they use tools. This sociotechnical system is therefore biased as the people in it are biased.

Ackerman’s line of argument suggests that the tools, if well designed, will create a “data domain” that might be interpeted in a biased way, but that this concern is separable from the design of the tools themselves.

A stronger (but then perhaps even harder to defend) argument would be that the tools themselves are designed in such a way that the data domain is biased.

Notably, the question of scientific objectivity depends on a rather complex and therefore obscure supply chain of hardware and software. Locating the bias in it must be extraordinarily difficult. In general, the solution to handling this complexity must be modularity and standardization: each component is responsible for something small and well understood, which provides a “data domain” available for downstream use. This is indeed what the API design of software packages is doing. The individual components are tested for reproducible performance and indeed are so robust that, like most infrastructure, we take them for granted.

The push for “reproducibility” in computational science is a further example of refinement of scientific instruments. Today, we see the effort to provide duplicable computational environments with Docker containers, with preserved random seeds, and appropriately versioned dependencies, so that the results of a particular scientific project are maintained despite the constant churn of software, hardware, and networks that undergird scientific communication and practice (let alone all the other communication and practice it undergirds).

The fetishization of technology today has many searching for the location of societal ills within the modules of this great machine. If society, running on this machine, has a problem, there must be a bug in it somewhere! But the modules are all very well tested. It is far more likely that the bug is in their composition. An integration error.

The solution (if there is a solution, and if there isn’t, why bother?) has to be to instrument the integration.

Academia vs. FOSS: The Good, The Bad, and the Ugly

Mel Chua has been pushing forward on the theme of FOSS culture in academia, and has gotten a lot of wonderful comments, many about why it’s not so simple to just port one culture over to the other. I want to try to compile items from Mel, comments on that post, and a few other sources. The question is: what are the salient differences between FOSS and academia?

I will proceed using the now-standard Spaghetti Western classification schema.

The Good

  • Universities tend to be more proactive about identifying and aiding newcomers that are struggling, as opposed to many FOSS projects that have high failure-and-dropout rates due to poorly designed scaffolding.
  • Academia is much more demographically inclusive. FOSS communities are notoriously imbalanced in terms of gender and race.

The Bad

  • The academic fear of having ones results scooped or stolen results in redundant, secrecy, and lonely effort. FOSS communities get around this by having good systems for attribution of incremental progress.
  • Despite scientific ideals, academic scientific research is getting less reproducible, and therefore less robust, because of closed code and data. FOSS work is often more reproducible (though not if its poorly documented).
  • Closed access academic journals hold many disciplines hostage by holding a monopoly on prestige. This is changing with the push for open access research, but this is still a significant issue. FOSS communities may care about community prestige, but often that prestige comes from community helpfulness or stake in a project. If metrics are used, they are often implicit ones extractable from the code repository itself, like Ohloh. Altmetrics are a solution to this problem.

The Ugly

  • In both FOSS and academia, a community of collaborators needs to form around shared interests and skills. But FOSS has come to exemplify the power of the distributed collaboration towards pragmatic goals. One is judged more by ones contributions than by ones academic pedigree, which means that FOSS does not have as much institutional gatekeeping.
  • Tenure committees look at papers published, not software developed. So there is little incentive for making robust software as part of the research process, however much that might allow reproducibility and encourage collaboration.
  • Since academics are often focused on “the frontier”, they don’t pay much attention to “building blocks”. Academic research culture tends to encourage this because it’s a race for discovery. FOSS regards care of the building blocks as a virtue and rewards the effort with stronger communities built on top of those blocks.
  • One reason for the difference between academia and FOSS is bandwidth. Since publications have page limits and are also the main means of academic communication, one wants to dedicate as much space as possible to juicy results at the expense of process documentation that would aid reproducibility. Since FOSS developed using digital communication tools with fewer constraints, it doesn’t have this problem. But academia doesn’t yet value contributions to this amorphous digital wealth of knowledge.

Have I left anything out?