Weapons of Math Destruction – the math behind the curtain

Published by Allan on

The dangers of game data – using math as a weapon

Recently a friend in the games industry shared an article about the dangers of in-game data collection and its intersection with game monetization. It was very timely, as I had just finished reading Weapons of Math Destruction, which focuses on an umbrella that these dangers fall under. I find the topic to be both important and urgent, so I wrote this post to share some insights and thoughts on these topics.

“Imagine micro-targeted cigarettes that could deliver the cigarette right in your fingers the minute you’re feeling the most vulnerable.”

Alex Champandard, The dangers of in-game data collection

The article on in-game data collection examines the idea of using data to weaponize our own minds against us. As scientists advance our understanding of the human brain, it creates a strong opportunity for businesses to take advantage of that understanding. When we know how addiction works and have the data to identify when people are most vulnerable to it, that can be weaponized against us. Imagine someone you know who has tried to quit smoking. That’s is a large undertaking on it’s own. In a connected world, businesses looking to improve the bottom line can make it even more challenging.

This all probably sounds like the setting of some dystopian sci-fi novel. Fortunately, we aren’t there yet, right? Well, about that…

Weapons of Math Destruction

Weapons of Math Destruction: How big data increases inequality and threatens democracy. Book by Cathy O'Neil.
Weapons of Math Destruction takes readers on a journey through life, exploring some of the mathematical models that manipulate our lives from behind the scenes. These models are invisible to us and the government representatives whose help we need to regulate them.

Last week I read a deeply revealing book on the topic of big data and statistical modeling. Weapons of Math Destruction by Cathy O’Neil pulls back the covers, revealing areas where math is doing harm. Mathematical models are working behind the scenes, increasing inequality and threatening our democracy.

Games are reaching a point O’Neil describes as a “weapon of math destruction”, or WMD for short. To qualify as a WMD, a model must:

  • Have negative impact on people, especially those who are vulnerable (damage)
  • Be a “black box” to those affected by it (opacity)
  • Impact a large number of people (scale)

WMDs are models that harm lots of people without their knowledge. While games are rapidly approaching this status, they haven’t achieved it yet. That is unfortunately not the case in many other areas of our lives. O’Neil takes readers on a hypothetical journey through life. In this journey, we learn about many places where WMDs are active:

  1. College admissions
  2. Online advertising
  3. Criminal justice (or in this case, the lack thereof)
  4. Getting a job
  5. Working a job
  6. Securing credit
  7. Getting insurance
  8. Politics and democracy

These are big topics where big data is causing big problems. Most models can’t target and predict all of us as individuals, so they do their work based on people like us. I’ll discuss this in depth here, but I recommend reading Weapons of Math Destruction if you want to learn more about the nuance and complexity of these systems. As a bit of a teaser, let’s look at getting car insurance.

WMDs take life from bad to worse

This may surprise you, but car insurance rates aren’t based on the skill and driving of the driver. Insurance companies don’t have detailed information about how we drive. Instead, they settle for proxy data that they do have. Data such as a credit score and zip code. In 2015 researchers at Consumer Reports uncovered the truth about car insurance. They found credit scores often impact insurance prices more than a history of drunk driving.

“credit scores often impact insurance prices more than a history of drunk driving.”

Life under siege by weapons of math destruction

This is terrible on its own, but the impact gets magnified by other WMDs. People struggling to make ends meet are charged more for insurance on the vehicle they need to get to work. Since many employers factor in credit scores, those who need better jobs the most struggle to get them. The jobs they work leverage WMDs to generate cost efficient work schedules. Those schedules are chaotic, wreaking havoc on the lives of the workers. When they go online to search for a way out they are targeted by advertisements for predatory loans and for profit colleges.

While looking for a way out, these advertisements lead them to more debt and worthless degrees. The poor can only afford to live in poorer zip codes that tend to have a history of more crime. History of crime is data used by models that send more police patrols. Increased patrols increase the records of trivial crimes like marijuana possession. When facing sentencing, a recidivism model tells the judge that poor people are more likely to re-offend, leading to longer sentences.

Getting kicked while you’re down doesn’t even begin to capture what WMDs can do. Simply for starting off with a worse initial condition, math is hard at work making it hard for people to improve their situation. It’s not bad for everyone though. Let’s look at how these models affect someone with money in a more comfortable situation.

Math makes the good life better

As you’ll see, people in a more comfortable situation have very different experiences with WMDs. Living in a rich postal code results in a lower car insurance rate. Access to SAT tutors and prep material makes admission to a high ranked college easier. College graduates, and especially those from high ranked school colleges have better job prospects. High credit scores improve their chance of receiving a hire decision. That same score helps them secure credit for a more expensive house in a low crime neighborhood. Since crime is low, local law enforcement patrols it less. Less patrols mean less records of common, low impact crimes like marijuana possession, jaywalking, and speeding. Low recorded crime leads to lighter sentences when arrests happen.

WMDs create a cycle that keeps giving to the “haves” and taking from the “have-nots”. It all happens in the background, out of sight and out of mind. When someone tries to find more information about how the WMDs work, they can’t. WMDs often exist as proprietary technology. Trade secrets that companies protect because the bottom line depends on them.

These issues are quite widespread in the US. We have some protections, but not enough. For example, Medical records are protected but psychological exam results are not. The companies that make fitness trackers are free to sell our data for profit, and their buyers can use that data for just about anything.

What can we do?

We need to regulate data and take a more critical look at the models we build. This is especially true for models that predict or interact with people. Modelers must consider how their models will be used. They must spend time on the ethical considerations that compete with the bottom line.

Disarm the weapons of math destruction

One of the qualities of a WMD is that it must cause harm. Changing that may be the easiest option for some WMDs. As an example, consider a recidivism WMD. Want to disarm it? Use it to allocate reintegration resources to those in greater need. It could be used to reduce recidivism instead of increasing sentence lengths. We can repurpose these models. They have great potential to help people who need it more efficiently. That’s a much better use of math!

Adopt and expand data regulations

The EU General Data Protection Regulation is a good start, but it’s not a final solution. We need to expand the classes of protected data. We need to regulate the ability to charge people based on proxy data – charging someone more for car insurance based on their zip code, while helpful to the bottom line, hurts individuals for something often beyond their control. 

Awareness and legislation are needed to make meaningful progress. Look at climate change. Now that more people understand the issue and its impact on their lives we are seeing behavior changes. Consider some of the actions spurred by climate change:

  • Bans on plastic straws and bags
  • Increased investment in renewable energy sources
  • The UN’s Paris Agreement

These actions are far from a solution, but they represent progress. Progress can feel slow at times, but slow is better than nothing at all. By continuing to discuss these issues, more people will become aware of them.

Through slow and steady progress, water carved out the Grand Canyon. Through slow and steady progress, we can make the changes necessary for a brighter tomorrow.

Did you enjoy this post? Share why or why not in the comments below! If you’d like to see more content based on books I read, you’re in luck! You can browse the full collection of them on my site.


I'm a Program Manager working at Microsoft in the Advanced Technology Group. Outside of work, my activities include reading, gaming, food, and building EventDNA, a startup focused on bringing people together. In my writing, I try to take a critical look at the world through my lens as a computer scientist and product person. All views expressed are solely my own, and do not represent EventDNA, Microsoft, or any other entity.


Leave a Reply