The political modeling that I talked about in the last post now affects most decisions that institutions make with respect to individuals.   This is nicely described in in the recent book Weapons of Math Destruction by Cathy O’Neil.   She has worked on these systems herself while at the hedge fund D. E. Shaw and at various e-commerce startups.

Clink for link to author blog

This modeling attempts to classify millions of people based on anything that can be gleaned about them online.  She has chapters on each of these categories of decisions:

• College Admissions – are driven by metrics related to the US News and World Reports rankings, which conveniently don’t include tuition.
• Sentencing and Parole – Who is likely to commit more crimes before and after jail time?
• Hiring – Study people’s social media, credit scores, and judicial records to see if they’re a good match for a firm.
• Firing – Teachers are especially closely judged these days because of right-wing opposition to the whole concept of public schools.   In particular, the No Child Left Behind Act almost forces teachers to be ranked and fired.   This has had the predictable consequences of teachers leaving low-performing school districts, skewing lessons towards the tests, and cheating.
• Borrowing – How are credit scores actually arrived upon?  FICO is actually a clear and straightforward metric, but lots of banks use mysterious e-scores these days.
• Insurance – Who gets covered and for how much?  The ACA forced consistent standards on the medical insurance industry, but that’s about to disappear.
• Voting – How can the news and advertising that people see be tuned to persuade them to vote one way or the other?  She actually discusses the work of Cambridge Analytica, which had a role in Trump’s victory, even though the book came out long before the election.

These decisions are largely made by computer these days because it’s cheap.   Interviewing students or borrowers or applicants takes real people with real skills, and that’s more expensive than just screening them by algorithm.   That means it gets done for the upper classes, who otherwise get annoyed by impersonal rejection, but not for the middle class and below.

But cheap methods are usually crummy, and that’s true here too.  They don’t have nearly enough statistical power to do a good job.   They’re using way too few data points (E.g. teacher evaluations are based on only 20 or 30 scores from wildly different children), are using proxies that have no real connection to what’s being decided, and have poor feedback paths to adjust the models.

Worse still, the methods are completely opaque to the people they are affecting, and often to the people using them.    An answer spits out, and there’s no recourse.  No one knows why they got turned down.   If they’re using a neural net, even the coders don’t know why it gives the answers it does.

Even worse still, the goal of the algorithm is entirely for the benefit of the organization running it.   No larger social goal can be applied, nor can any larger sense of fairness.  Thus the algorithm can easily cause death spirals. E.g. by denying mortgages to certain neighborhoods, the area declines, making it less attractive for investment, causing further declines.  By denying people bail or parole, whole classes of people can be put in decline.  The algorithm may optimize the short-term profit of the people running it, but is too mysterious to service long-term goals even for them, much less society as a whole.

She contrasts this with player evaluations in major league sports.  The statistics about a player’s performance are all publicly known, and are plentiful if they’ve been in the game for any time.  They are directly related to the main question – how much will this player help the team win?   The model can be constantly run to verify its predictions, and adjusted when wrong.   But if you have a model for who makes a good hire, you really only get to see a little about who gets picked, and then only at infrequent reviews.  You don’t learn anything about the people rejected.

So what is to be done?  Her suggestions don’t seem that helpful to me.  She focuses on an ethics code for programmers of such algorithms.    That’s been a valuable approach in civil engineering, where people really are conscious that their work can kill when it fails.  Few other engineering disciplines insist on this, though.  The connection between one’s work and its consequences is much more remote in programming than it is in construction.

It’s better to have public and independent inspections.   That’s how bridges get certified.   It’s coming to be how components get certified in cars and airplanes.  An outside party reviews the design process and the safety behavior of a device and gives it a rating.

That’s hard for big software systems like these, especially since they’re considered to be a business advantage.  What people can do is test the system with simulated applications.  She describes how researchers can create fake online personas to see how their social media gets steered, or their search results, or their college and loan applications.

The Big Data companies like Facebook and Google hate this, though, and do everything they can to prevent it.  They don’t want people to know how they’re being judged, for fear that users will game the results.   There’ll be an arms race between the parties trying to understand what Big Data is doing to society and the increasingly malevolent firms themselves.

Anyway, the book as a whole is clearly written and thorough.  Her blog is excellent too!   It’s a good overview of a problem that will only get worse.