This is the first part in a 3-part series I am writing on the concept of performativity and how it interacts with machine learning and computer algorithms.
In this first part, we’ll explore how algorithms enact performativity in the world. In Part 2, we’ll unpack how embedded worldviews shape their definition of ‘best,’ and in Part 3, we’ll look at how we might resist or redirect these feedback loops.
We live in a modern world where, in every aspect of life, progressively more sophisticated algorithms and machine learning algorithms are used to make real-world decisions. From the YouTube algorithm deciding which videos are shown and to whom, or in the case of the article you can read here, where predictive policing leads to empirical runaway feedback loops. In this study, researchers look at a common tool currently used by local law enforcement agencies to predict where to deploy officers, in an attempt to anticipate the trends in crime in the city. The system that local police agencies use for this is called ‘PredPol’, and this system operates by looking at reported incidences of crime and arrest counts from a specific neighborhood. It is then updated with new data in batches, called batch analysis. They demonstrate how systems such as this are susceptible to runaway feedback loops, where police are repeatedly sent out to the same neighborhoods, no matter what the actual crime rates are in the neighborhoods that you’re monitoring. Some key results I found interesting from this study are that, even if all the neighborhoods are assumed to have equal crime rates (or nearly identical), this system still will have a runaway feedback loop and send police disproportionately to one neighborhood. They successfully found a workaround in this system using statistical modeling to reduce the effects of the feedback loop. In this exact case, they needed to reduce the amount of significance given to the arrest counts for the neighborhood. Doing this allowed them to successfully predict crime rates for simulated neighborhoods instead of experiencing runaway feedback loops.
This is not simply a misjudgment by the algorithm, but I would describe it as a modern example of performativity. This concept originates in linguistic philosophy and has spread to many other areas of study. For this article’s purposes, we will consider performativity to be “Generally when certain actions produce effects or bring about a new state of affairs”. We will also examine how the optimization of these algorithms can lead to behaviors that reinforce the algorithm’s model. We will pay very close attention to how “optimization” is generally synonymous with “best” in these algorithms’ eyes. The “optimal” outcome is the best only according to the goals, constraints, and worldview embedded into the model. This reminds me of insights from the philosopher Immanuel Kant, which is the idea that our minds bring assumptions to the data, and to what we perceive to be the best for the situation. These innate assumptions and worldviews are built into the models we construct.
Let’s look at performativity a bit more before we move on. It has its origins in linguistics, and was not originated by, but was popularized as a concept by John Austin in the 1950s. Originally, Austin was looking at statements in society and everyday life that, once uttered, would, merely by the words having been spoken, change some state of affairs. An easy example of this is when someone dubs the name of a ship, “I name this ship The Titanic.” By announcing this, the ship is then named so. This concept was followed by more philosophers in linguistics and critical theory, like John Searle and Jean-François Lyotard. Judith Butler’s work on gender identity using performativity gained a lot of traction. I think some of the best examples of modern performativity would be to look at economics. This generally outright states that professionals and popularizers affect the phenomena they purport to describe, causing a feedback loop of new results that were based on the model feeding into the model. This article does a great job of outlining this example in economics in great detail. They claim performativity occurs when the act of modeling or predicting alters the behavior of the subjects being measured, aligning them more closely with the model itself.
This feedback loop, already visible in finance, education, and governance, becomes especially powerful when embedded in AI-driven optimization systems. The authors have many great examples, including my personal favorite being retail inventory algorithms. Large retailers use predictive algorithms to stock products based on expected demand. As stores adjust stock based on the algorithm, consumer purchasing patterns adapt to what’s available, reinforcing the algorithm’s original prediction. As a result of performativity, these systems shape the range of actions that are perceived as “possible” or “worthwhile”, based on the algorithm’s output. This has real-world effects on people like the MMO video game Old School RuneScape players, and they speak about how they actively feel like they are wasting time not doing methods of training deemed “efficient”. To other, more concrete examples, like a research group not being selected by an algorithm to receive funding for cancer research. This feedback loop can affect both the seemingly trivial and the critically important.
Whether it’s a police patrol route, the products on a store shelf, or even the way we train in a video game, these models don’t just reflect the world; they make the world. And as we’ll see next, the way an algorithm defines ‘best’ can quietly set the boundaries for what is even possible.
If algorithms can reshape the world to match their internal logic, the next question becomes: whose logic is it, and what counts as ‘best’ in that worldview?
Part 2 next week
The Quiet Syntax 