Nothing is capable of making mistakes more invisibly or at such scale as AI / Machine Learning

Last month, in a paper presented at an Association for Computing Machinery Conference, computer scientists Hundt et al provided a strong reminder that it is very hard to stop AI, or Machine Learning, from becoming a very dangerous mistake, or an organizational weapon. Quoting from their abstract:

"Stereotypes, bias, and discrimination have been extensively documented in Machine Learning (ML) methods such as Computer Vision (CV), Natural Language Processing (NLP), or both, in the case of large image and caption models such as OpenAI CLIP [14]. In this paper, we evaluate how ML bias manifests in robots that physically and autonomously act within the world. We audit one of several recently published CLIP-powered robotic manipulation methods, presenting it with objects that have pictures of human faces on the surface which vary across race and gender, alongside task descriptions that contain terms associated with common stereotypes. Our experiments definitively show robots acting out toxic stereotypes with respect to gender, race, and scientifically-discredited physiognomy, at scale."

Robots Enact Malignant Stereotypes

FAccT '22: 2022 ACM Conference on Fairness, Accountability, and Transparency, June 2022 Pages 743–75 https://doi.org/10.1145/3531146.35331

This is not a new realization. In a previous blog, I discussed how mathematician and machine learning expert Cathy O'Neil described this risk very clearly in the book Weapons of Math Destruction (2016).

Owners and builders of AI / Machine Learning systems need to keep in mind of a simple fact. These systems contain a correlative model which is only as good or as relevant as the training data someone chooses to give them and the training labels someone selects for them.

They are not intelligent. They don't experience the world like you or I. They don't relate to what it feels like to be hungry or eat an apple. In technical terms, their understanding of their world is not grounded in a human reality. Worse, for systems that learn from natural language data, which is not real-world sensory data, their understanding is not grounded in reality at all (natural language data is extremely narrow and edited - it only reveals what the writer or speaker believed was important, and usually leaves unsaid the shared assumptions of sender and receiver).

So what can this produce? It can easily result in models that may lock-on to spurious shallow correlations based on tokens that happen to be present in the training sample, and onto unintended or malicious statistical biases in the training labels.

It also can result in models that work very badly when your data is not drawn from the same statistical population as the training data sample, or when you would not have agreed with all the labelling decisions.

The most obvious result is systematic error (as opposed to random error) in the outputs, and these can only be discovered by regularly performing your own validation of the outputs.

Pragmatically, the real questions for a prospective user of such as system are:

• Was the training data sample statistically representative of the data population you will be using?

• Did the training data contain tokens which directly capture the measurables you want to attend to in your system, or did they use so-called proxy variables, which are often conveniently available, but don't directly measure the desired quantities?

• Were the training labels assigned in a manner optimal to your real needs from the system, or did they contain unknown biases or cognitive shortcuts (such as prejudice); or were the human labelers simply wrong or naive?

• Can you afford to perform your own regular statistical validation of the outputs from this automated system?

A recurrent problem I see in the literature describing Machine Learning models is that the measured accuracy of the prediction of human specified labels from untrained data is the only validation required.

I rarely see any analysis of which data variables the model is relying on most. In fact, one of the hallmarks of deep learning is the claim that these models automatically weight and select the important input data dimensions.

I think this is a serious vulnerability for the method and can lead to the model relying on discredited proxy variables, or just systematically poor and naive labeling.

Given that the selection of meaningful and fundamental input variables and outcome variables in a data model is a key task of a trained expert (economist, business analyst, scientist, etc) who is familiar with the data domain, the rejection of human experience in the selection of which input and output variables are meaningful is a short-cut which should be questioned much more assertively.

I also rarely see discussion of the scope or generality of a trained model. In text and natural language modelling, there is a very frequent and inexplicable assumption that words mean the same thing everywhere, for all speakers and all times. In other words, you can train a language model once, so long as you use a very large amount of text, and then it will work for anyone, in any specific sub-culture of communicators dealing with any topic at any time. I have never seen this to be true over 22 years of applied text analysis. Models of word meaning are quite variable, and models of semantics (i.e graph of related ideas from a situation) even more so. You just have to imagine the transcript of a brain surgery conference or a conversation in a public bar to get the idea.

There is a growing use of tools to tweak or nudge a trained model using your own language data, but I cannot see why one would want to rely at all on a model trained using data significantly different from your own.

Leximancer language models are almost always built from your own data at the time you need to analyse it, and they don't require human labeling (i.e they are unsupervised by default). The nice thing is that you can easily step in and train your own variables on your own data (using semi-supervised learning).

And you can always see what data is driving the model outcomes, and how the deep learning network is defined.