Cathy O’Neil’s Weapons of Math Destruction doesn’t break new ground in its discussion of how computer algorithms discriminate against the poor, minorities and women, making them pay more for loans, shutting them out of jobs, and disrupting their work schedules. Just about everything she writes about we’ve seen in the pages of major newspapers and serious magazines.
But O’Neil puts it all together in language we can understand and with the rigor of a mathematician who has actually delved into the various assumptions and computations hidden within the digital black boxes that companies use to sort resumes, banks use to give loans, employers use to schedule employees and virtually every consumer company uses to target customers with products and services.
O’Neil does not advocate doing away with all the mathematical models that permeate contemporary American society, only those that threaten our social fabric because of hidden prejudices built into the algorithm or those that purposely exploit or deceive people.
Take the development of U.S. News Report’s top college rankings, which over the past thirty years has engendered a “keep up with the Joneses” competition among status seeking parents, while dramatically changing how universities approach their own development and improvement, pandering to the rating instead of the educational mission of the institution. It has also created a new industry of college selection advisors to help rich and middle class parents get their children into the highest-ranking schools. O’Neil reports that the original mathematical model used as its measures of success those variables in which the schools thought to be traditionally the best had excelled, such as contributions by alumni. Their selection of what to measure not only favored Harvard, Yale, Princeton, Stanford and the other universities already entrenched at the top of the pecking order, it also led to such obvious distortions as small liberal arts colleges for the wealthy achieving a higher rating than state universities with far-ranging research capabilities and an economically diverse student base such as Washington, Texas, Wisconsin, North Carolina and the University of California Berkeley and Los Angeles campuses.
It’s amazing that one misshapen computer model could do so much damage. But O’Neil analyses a number of such monstrosities, such as the biased models by which school districts evaluate teachers, banks decide which financial services to offer and how to price them, employers analyze resumes without looking at them to decide whom to hire and political campaigns select which messages to send to individual voters. O’Neil calls these models “Weapons of Math Destruction,” or WMD, both clever and accurate.
Many of the problems caused by WMD stem from substituting a simple measurement for a complicated situation, for example when teacher evaluation models substitute test scores for in-class performance to measure teachers’ competence or employment models use credit scores to determine a potential employee’s stability. Another major problem is that many of the models have as their sole purpose the maximizing of profit, regardless of what that means to customers or employees, such as job scheduling models that make employees work split shifts, add or cancel their work hours before the shift begins, and prevent their total hours from exceeding the minimums for receiving benefits. The employer makes more money, while financially strapped employees have to deal with juggling childcare, medical appointments and other aspects of daily life. Then there are the models that instantaneously analyze your Internet browsing history as soon as you get on the website of a financial institution, telling the institution whether to offer you a high or low rate on loans and insurance, based on your “risk.” High risk in this case serves as a euphemism for poor and often, minority.
An anecdote O’Neil tells near the end of the book is particularly scary because it reflects the anti-science, anti-fact bias shared by many corporations and politicians. Despite years of work as a successful mathematician in the private sector, in 2013 O’Neil took an unpaid internship in New York City’s Departments of Housing and Human Service. She was interested in building mathematical models that help, not hurt society. The issue was homelessness. Her team looked over masses of data to figure out what factors led people into homeless shelters and what factors led them to leave and stay out for good. One of her colleagues discovered that one group of homeless families left shelters never to return—those who obtained vouchers for housing under a federal housing program called Section 8.
Ooopsy! As it turns out, then NY Mayor Michael Bloomberg, the data king who had made billions of dollars supplying the financial industry with information, didn’t like Section 8 vouchers and had instituted a highly publicized new program called Advantage, which limited Section 8 subsidies to three years. As O’Neil writes, “The ideas was that the looming expiration of benefits would push poor people to make more money and pay their own way.” Yeah, right—they’d finally stop working as a fast food cashier and take a job as a Wall Street lawyer! Of course with rents booming in the Big Apple, the opposite was happening: people lost their vouchers after three years and ended up in homeless shelters.
The Bloomberg administration did not welcome the researcher’s finding and evidently ignored it in future planning. What the Bloomberg Administration wanted to believe literally trumped (pun intended!) what the facts were suggesting: that the cure for homelessness was not unfettered capitalism but providing a helping hand. Ignoring what the research proved corrupted Bloomberg’s approach to reducing homelessness as much as the current Trump administration’s approach to the environment, government regulation, taxation and education is corrupted by its failure to follow the facts. Corruption and manipulation lie at the heart of what caused institutions to create and apply WMDs.
I vividly remember an example of the corruption of ignoring facts I experienced when I worked for a large public relations agency in 1987. I returned from a Conference Board seminar with a study about the way corporations would employ agencies in the 1990s. The new approach to agency use would make it harder for agencies to make money and force them to engage in more competitions with not just other large agencies, but small boutique firms that specialized in one kind of PR. As soon as I completed my presentation to the staff of our office, our general manager got up and said I was wrong and outlined a rosy view based on no research whatsoever. Of course, the predictions presented at the Conference Board seminar turned out to be right on the money.
Thus, while we must beware weapons of math destruction and devise industry standards for both developing mathematical models and regulating their use, the greater problem is the age-old one best expressed in a quote often attributed to Mark Twain that Figures never lie, but liars figure.