In an interview, computer science professor Ulrike von Luxburg talks about the possibilities and difficulties of trimming machine learning systems for fairness – and why she is convinced that people, not machines, should decide on certain issues.
Machine Learning for Science: recently, reports of algorithmic discrimination have been accumulating. Seemingly objective, algorithm-based systems decide to the disadvantage of individuals and thus turn out to be unfair. We're talking about facial recognition programs that simply don't recognize black people, for example, or programs that prescreen applications for a particular job and rate men's resumes better than those of women. The societal demand on developers, as well as researchers, is often: correct the! Is it as simple as that?
Ulrike von Luxburg: We push a button and make the algorithm fair – it's not that simple. Machine learning isn't just the one algorithm I use, after all. But actually machine learning is quite a long pipeline.
How do you mean that?
It could already be in the data, who collected it, how it is labeled. It could also be due to the definition of the groups, i.e. the question of who I have to be fair to, and then the algorithm comes first. And in this whole pipeline, you have to think about fairness as well.
Let's go through this pipeline, the different steps, together, shall we?. So in the beginning there is the data. The data with which the algorithm is trained, based on which it learns to make decisions.
"Often, the data was not compiled for the purpose for which it is then used in machine learning."
Right here is already where the first "bias" comes in, that is the first bias or prejudice. In many of the applications that have been discussed publicly, where black people have fared poorly, things are already going wrong at this point. There were far too few images of black people in the data. By now, I think everyone realizes that if they do facial recognition, people with different skin colors must be well represented in their data set. Or in the case of resumes: If mostly men have been hired in the past, then that's naturally in the data, and a system trained on that will try to mimic that behavior. Often the data was not even compiled for the purpose it is then used for in machine learning. So an important question is: where does the data come from and who chose it? Are they specifically collected or are they simply crawled from the internet? And who rates them, labels them?
The labeling, i.e. the classification into categories, is often done by crowdworkers, i.e. people who take on smaller jobs as freelancers via Internet platforms, more or less on demand.
And this is how, for example, tools are created that are supposed to evaluate the attractiveness of a person in a photo with the help of machine learning. Labeling is then done by 25-year-old men – crowdworkers are mostly male and young – and rate how attractive the people in the photos are. Such a dataset is then a priori "biased" and primarily reflects the preferences of the crowdworkers involved.
Let's move on to the next step, the question of who you want or should be fair to.
"For any particular application, one must first consider what "fair" even means in this context."
Ulrike von Luxburg has been co-leading the Cluster of Excellence "Machine Learning" with Philipp Berens since 2019 © SOPHIA CARRARA/UNIVERSITY OF TuBINGEN
First, there is the definition of fairness: which groups do I want to be fair to? Women compared to men? Blacks vs. whites? For me to make an algorithm fair, I have to "tell" it that up front, and the more groups I name, the harder it gets. And then comes the fairness term itself. For any particular application, one must first consider what fair even means in this context. Here are a few standard cases. One, for example, is "demographic parity" or "balanced proportions". For example, one might say that a university should reflect the ratio of males to females in student admissions as it does in the population, i.e., admit about half women and half men.
People only look at the absolute numbers, not the qualifications of the people. Another fairness term would be "equalized odds" or "equalized opportunity", which can be translated as equal opportunity. If we stay with the university admissions example, this would mean: If you have the same aptitude, you – no matter if you are a woman or a man, if you are black or white – should be admitted to this study program. You can't meet all the notions of fairness at the same time, you have to pick one.
It all sounds like machine learning processes can be trimmed for fairness quite well, as long as you're clear about what fairness means. Where is the catch?
The moment I want to establish fairness, other things go down the drain. If I want more fairness, for example, the accuracy of the predictions goes down.
What does that mean exactly??
This may sound a bit abstract. But it simply means that if, for example, we are talking about granting loans, and we want an equal number of whites and blacks or men and women to get a loan, then I might also give a loan to people who might not be able to pay it back. But in the end the money has to come from somewhere. The bank or the customers or society would then have to jointly pay for the lost money. That is, there is a cost associated with it. The question then becomes: How much is fairness worth to us??
After the question of data collection and the definition of the concept of fairness, we have now arrived at the algorithm.
There are two goals for the algorithm: On the one hand, it should be fair, on the other hand, it should be accurate. To stay with our example: In spite of the defined fairness criteria, the algorithm should pick out the candidates for a loan who will also pay off the loan. Now I have to solve this trade-off between fairness and the actual goal of the algorithm. Here I now have a set screw: How much fairness do I want, how much accuracy?? As a bank, for example, I can decide to give ten percent of my loans to needy people. But I can also decide it should only be five percent. Depending on how I decide, fairness goes up or down. At the same time, depending on this, the accuracy – and thus ultimately the costs incurred – goes up and down.
Let's say I actually decide as a university to use a start-up's algorithm to automate student selection and save on staff and costs. Then of course I would like to know if this algorithm makes a reasonably fair selection. But how algorithms are constructed is usually a trade secret that companies rarely disclose.
That's a question that I find exciting: How could a state try to certify something like this?? If you think about the future, there are many start-ups that are launching algorithms, and they also want to be able to say, "We do this well. And they would like, for example, something like a TuV seal on their website that says: "Tested by the Federal Data Protection Office: Is fair". Or at the very least, "Fairly within reason.". But how would this look like? How would one define a kind of minimum standard that is testable afterwards without having to disclose the algorithm?? I often discuss this with my staff, but we don't have a ready-made solution either.
In her research, Ulrike von Luxburg is concerned with understanding machine learning algorithms from a theoretical point of view. © SOPHIA CARRARA/UNIVERSITY OF TuBINGEN
In your opinion, how should a society, should a state position itself as long as there is no "TuV" for algorithms? The only remaining option is to declare sensitive areas, where discriminatory decisions would have far-reaching consequences, as a taboo zone for algorithms?
I think there are actually areas where I would not want to have such a system for ethical reasons. When it comes to life-relevant decisions like going to jail or not, taking a child away from someone or not – you can't just delegate that responsibility to an algorithm.
One might argue that the algorithm-based system does not necessarily have to make the decision. It could also just be an assistance system that makes suggestions to us humans.
You hear this argument again and again, but in practice it often does not work out. The judge, who is under time pressure anyway, will not constantly decide against the assistance system. The tendency will always be to follow the recommendations of the system. But there are other areas where I would say: There are other areas where I would say that machine learning systems can do some good. Medicine is a case in point: an assistance system that makes suggestions for diagnoses or medications to take. That's where I would say if it's done well, maybe the benefit is greater than the harm after all. I certainly see potential there in the near future.
"It could be that machine learning systems are better or fairer than humans in some places."
In general, you will have to get used to the idea that these systems are not perfect and that you simply have to deal with this fact. But it could be that they are better than humans or fairer than humans in some places. After all, one thing is clear: even human decision makers aren't always fair and have biases that influence their decisions. The difference is perhaps: we now have methods at hand to evaluate the fairness or accuracy of an algorithm, but just as well the fairness or accuracy of human decision makers. The comparison between the two could sometimes favor people and sometimes favor machines, depending on the application.