Machine Learning and Artificial Intelligence Can’t Predict the Future

What Madden NFL can teach us about cybersecurity

Last year, in the wake of much talk about the wonders of Machine Learning (ML) and Artificial Intelligence (AI), I was looking for a way to assess how effective these kinds of technologies are at predicting the future. Knowing the popularity of the Electronic Arts’ Madden NFL video game franchise, I thought an analysis of the accuracy of that technology in predicting the NFL season would be a relatable way to see how accurate ML/AI is at predicting future events when fed a significant amount of normalized data within specific guardrails. I released those results and decided to look back on this most recent season to see if Madden NFL fared any better.

Machine learning and artificial intelligence predictions

For nearly two decades there has been a steady drumbeat around how ML and AI will solve all the world’s ills. From military intelligence to cybersecurity, experts continue to say that machines will replace human decision-making. And now the latest technology to capture attention is ChatGPT. Depending on who you ask, ChatGPT is either the replacement for white hat coders or the tool that will make threat actors unstoppable…or both!

But while there is great value in these tools as enablers, we are far from automating away our problems. Many people think AI will lead us to identify the “unknown unknowns,” yet most experts I’ve spoken with don’t believe this is possible. In fact, the U.S. government has been trying to find a way to automate Intelligence for decades so they can reduce the dependency – and labor costs – of a massive workforce. Funnily, despite that desire and seemingly endless resources to pursue ML and AI, the number of intelligence analysts in military and government roles has not diminished.

As someone with only a passing knowledge of ML and AI, and not the scientific background to understand all the nuances, I’ve spoken with experts in the field who almost universally say these technologies have incredible promise but are still more hype than reality today. Knowing this, I wondered if there was a tool (that I could understand) that contained a robust amount of data used for modeling future events and a means for judging its accuracy in doing so. The Electronic Arts (EA) Sports video game franchise Madden NFL, which is used annually to predict the entire NFL season, was the perfect tool for me since it’s loaded with data, includes ML and AI, and is associated with a topic I understand and enjoy: football.

Madden NFL data and predictions

To get a baseline for performance going into an NFL season, EA Sports creates annual ratings for 2,600+ players across 53 criteria and applies significant weighting to the ratings to create overall player ratings. These ratings are based on hundreds (if not thousands) of hours of analysis that span up to several years of players’ careers. The data is impressive, and for the millions of fans of the game, it is the foundation for what is considered the pinnacle of sports-based video game franchises. Despite that, Madden NFL has not fared well when trying to accurately predict an entire NFL season. Focusing on the last five seasons, Madden NFL predicted three of the 10 conference champions (30%) and one Super Bowl champion (20%). While those numbers may not sound bad, consider that Madden NFL also predicted four of the cumulative 10 Super Bowl slots (40%) would go to teams that did not even qualify for the postseason. Those results do not instill great confidence in the ability to predict the future.

For the recently concluded 2022 season, Madden NFL predicted that the New Orleans Saints would defeat the Kansas City Chiefs to win Super Bowl LVII. Madden not only correctly predicted the Chiefs' upcoming Super Bowl appearance, but it also got quite close to predicting their final record, The 13-4 prediction was only one game off of the Chiefs’ actual record of 14-3. Interestingly, this was the fourth consecutive year that Madden NFL predicted the Chiefs to represent the AFC in the Super Bowl…and three of those have been correct. While a 75% success rate is accurate, it’s worth noting that Las Vegas oddsmakers also picked Kansas City to reach the Super Bowl in each of the last four seasons.

While Madden NFL was right on track with the AFC champions, things on the NFC side were not nearly as impressive. The New Orleans Saints, the Madden NFL choice to reach the Super Bowl for the NFC, finished the season with 7 wins as compared to 10 losses. They failed to even qualify for the postseason despite playing in a historically bad division where no team had a winning record. Part of the disappointment to the Saints’ season can be placed on their projected starting quarterback sustaining a season-ending injury, but that just speaks to the difficulties in trying to predict the future.

If you’re asking yourselves how Madden NFL saw the future for the true NFC champions, the Philadelphia Eagles were predicted to be a wild card playoff team that did not even advance to the NFC title game. Instead, Madden NFL had the Dallas Cowboys posting a 12-5 record (which they did!) before falling to New Orleans in that fictional NFC Championship game.

Looking beyond predictions for the conference championship games (where Madden NFL’s accuracy was 25%) and the Super Bowl (where Madden NFL was 50% accurate), the EA engine fared poorly at picking the entire postseason field. The simulation only accurately predicted two division winners (25%) and six of the fourteen teams that qualified for the postseason (43%). 

Madden NFL had high hopes that the Chicago Bears would post a 9-8 record; far better than their actual mark of 3-14. But that pales in comparison to the Madden NFL prediction of a 16-1 Indianapolis Colts team that would be upset in the AFC Championship by Kansas City. In reality, the Colts’ season was an unmitigated disaster. After starting 3-5-1, they fired their head coach (Frank Reich) and replaced him with Jeff Saturday…who led the team to a 1-7 record the rest of the way. Needless to say, Madden NFL’s prediction of Frank Reich as NFL Coach of the Year did not come true. The reverse of Madden NFL’s overly optimistic outlook for Indianapolis and Chicago was the pessimism around the Los Angeles Chargers. While Madden NFL expected the Chargers to post a 4-13 record, the true result was a respectable 10-7 mark that included a playoff appearance.

Madden expected big years from Trey Lance (QB, San Francisco) and Cooper Kupp (WR, Los Angeles Rams), but both were plagued by injuries that derailed those plans. Baker Mayfield was expected to post big numbers as the undisputed leader of the Carolina Panthers (he was actually replaced and released) and Sam Darnold was oddly predicted to be a start after a trade to Pittsburgh. In reality, he was in and out of the Carolina lineup and has little job security now. Other impressive performers in the Madden NFL simulation included Matt Ryan (QB, Indianapolis) and Carson Wentz (QB, Washington). Neither had even average seasons and both will soon be unemployed.

One exciting prediction Madden NFL seems to have gotten right was the retirement of Tom Brady, which he announced via social media on 1 February. If Brady stays retired this time, it would be the most impressive - albeit random - prediction Madden NFL had for this season.

Madden NFL and the Super Bowl

For those who argue that predicting an entire NFL season is too complicated, let’s expand on what we learned last year. In just focusing on one game - the Super Bowl - Madden NFL has accurately predicted the winner just four out of the last ten times (40%). For comparison, an elephant in El Paso, TX has reportedly predicted eight of the last eleven Super Bowls correctly (73%). Was Madden NFL’s algorithm weighted to favor underdogs? No. The four correct predictions included two favored teams and two underdogs. The six incorrect predictions were also evenly split between favorites and underdogs (3-3).

If you’re wondering who Madden NFL picked to win this year’s Super Bowl, that information hasn’t been released yet. But, when choosing whether to trust the Super Bowl prediction of Madden NFL or the El Paso Zoo elephant, I gotta go with Savannah…the elephant.

What does this have to do with cybersecurity?

EA Sports, in cooperation with the NFL and NFL Players Association, collects a remarkable amount of data on each player and scheme (offensive and defensive) in today’s NFL. Capitalizing on that input, simulations using Madden NFL apply their machine learning algorithms to predict the result of thousands of plays across hundreds of games in a given season. It sounds nearly impossible, and the results year after year seem to confirm that fear. Madden NFL has not been able to predict the outcome of an NFL season in meaningful ways and hasn’t gotten more accurate despite at least a decade of improvements. 

The monumental task of attempting to predict so many variables over a long period of time - as these Madden NFL simulations are asked to do - is not entirely different from attempting to predict events in cybersecurity. We track hundreds of threat actors and groups using at least as many Tactics, Techniques, and Procedures (TTPs) — including variants — against an ever-growing list of security stacks and configurations. Worse yet, unlike with Madden NFL, our assessments of threat actors, groups, and TTPs is largely based on piecing together what we know and making analytic judgments. That’s a far cry from having the validated measurables (height, weight, speed, and weight room testing, etc.) on NFL players that EA Sports has to work with.

Hype vs. Hyperbole

Why do we continue to hear the drumbeat of ML and AI as the solution to all of our cybersecurity woes? Why do so many security pros, technologists, and vendors proclaim that  with enough machines and enough data, we can automate ourselves to safety without the need for people? It’s not because the claims are true today. Rather, it’s because cybersecurity professionals desperately want to believe in an easy future compared to the challenges we all face today. Companies are enticed by promises of getting equal or better security outcomes with lower labor costs (when machines replace people). Knowing this, some unscrupulous suppliers will overstate the power of their technology to build their businesses and increase their profits. I’ve even heard rumors of vendors who claim to have ML or AI when they are actually dependent on large teams of inexpensive labor doing data entry at scale. When was the last time anyone asked to examine a company’s claimed technology? How many customers would even understand the algorithms if they had access to them?

This isn’t to say there is no value to these advanced technologies. While AI still appears to be more of a (promising) theory than a reality (see: self-driving cars), we already use ML to identify malicious objects and images, process more data than even a large team of people could accomplish in a reasonable timeframe, disrupt threat actors who bypass text analysis, and give people a fighting chance to identify, prioritize and act on the most significant threats.

Given the right training data, and enough time, ML is impressive. It just can’t accurately predict the future, which is hardly a criticism. The key to proactive cybersecurity continues to be the combination of superior access to data, information, and intelligence; powerful tools (including ML) to normalize, deduplicate, categorize, and prioritize that content at scale; and experts to apply Intelligence tradecraft, experience, and intuition.

I honestly don’t know if there will come a day when ML and AI fulfill the bold promises of increased security and reduced costs through the replacement of people. But I can confidently say that today is not that day…and I’m willing to bet that tomorrow won’t be, either.