I recently started reading The Phoenix Project, a fictional tale about DevOps. I got several chapters in and felt like stopping, not because it isn’t an interesting story, or poorly-written. I think it was because it was hitting close to home, having expereienced some of the character stereotypes and situations personally.

I’m still not even half way through, but I’m pressing forward, and I think it’s starting to pay off.

I’ve been seeing what I consider a disturbing trend in thinking in software development circles: this idea of “algorithmic bias,” or, the idea that algorithms are biased for or against certain groups or demographics. To be clear, it’s not the idea that some algorithms are biased, but this notion that the algorithm by itself is biased, that, somehow, the computer code is biased apart from the design intentions and biases of the programmer who wrote it.

I stumbled on an article on Medium that attempted to substantiate this idea. The article was poorly written, as demonstrated below.

The first is a “basic example” of two programmers tasked with writing an algorithm to sort baskets of fruit.

Given the same task and data, the two people are likely to have different results. Perhaps one person believes the primary indicator of a fruit’s quality is brightness of color. That person may sort the fruit based on how vibrant it is, even though not all fruits are brightly colored; that person would send strawberries to the market and melons to the jam factory. Meanwhile, the other person might believe that unblemished fruit is the best quality, even though fruits with protective rinds might look scruffy on the outside, but are perfectly fine on the inside; that person could send unripe strawberries to the market and ripe melons or bananas to the jam factory. These different, yet similarly logical and evenly applied criteria, will result in two different outcomes for the same basket of fruit.

What? A programmer worth his/her salt is not going to approach writing this kind of algorithm based on their subjective beliefs about what determines a fruit’s quality. You might like green bananas, while I may prefer ripe, spotted bananas, but that has nothing to do with the quality of those bananas. Bruises, gashes, and other things are more indicative of a lower quality banana, and are more objective. The authors essentially present two poorly-designed, poorly-written, subjective algorithms as demonstrations of the algorithm’s bias.

They continue with:

Makoto Koike wanted to apply machine learning to help his mother more efficiently sort the cucumbers from her farm. Rather than asking his mother to define the features she used for sorting, Koike tasked her with sorting a bunch of cucumbers. He then optically scanned the cucumbers and used computer software to identify the common traits and then built an algorithm to replicate her work. That algorithm’s understanding of what makes a “good” cucumber was based on Koike’s mother’s interpretation and intuition.

Again, the authors present an algorithm designer approaching a problem from the wrong angle, and then designing something that is inherently deficient, without even bothering to consult the person for whom the algorithm is written, and that’s supposed to be a convincing argument that the algorithm is biased?

Lastly:

Ten years ago, the Federal Reserve Board, under direction from Congress, evaluated whether credit scoring methods were discriminatory. In its report to Congress, the Federal Reserve Board revealed a strong correlation between credit scores and race and other demographic indicators, even though “credit characteristics included in credit history scoring models do not serve as substitutes, or proxies, for race, ethnicity, or sex.” Nevertheless, credit scoring models disadvantage some segments of the population more than others. Immigrants tend to have lower credit scores, for example — not because scoring algorithms are trained to assign immigrants lower credit scores, but because length of credit history weighs heavily in scoring models, and recent immigrants will have had less time to develop their credit histories.

Of course the algorithm discriminates against immigrants who have no credit history. That’s the way it’s supposed to work! Why should an immigrant be granted a handicap in the credit rating system? Are immigrants inherently lower credit risks than non-immigrants?

This type of thinking has become pervasive in the software industry, and wider. Don Quixotes tilting at windmills; social justice warriors searching for and finding problems in every corner and crevice of the world. Discrimination in and of itself is not evil, but the authors clearly don’t understand this. There are proper applications of it, and improper (and far too many, unfortunately). The examples cited here are weak or inappropriate, and all the authors demonstrate is the inability to understand the problem, how to argue about it, and how to fix it.