When AI Speaks, the Next Phishing Innovation May Be Listening

IMDb describes episodes of Black Mirror, the British sci-fi television show available on Netflix, as “set in a world only minutes from our own . . . where humanity’s greatest innovations and darkest instincts collide.” It’s unsettling. It scars precisely because each story is seated upon a very realistic premise.

Take ‘Be Right Back,’ the first episode of season 2. A woman’s (Martha) beloved, whimsical young husband (Ash) is killed in a car crash. An artificial intelligence (AI) service trained on Ash’s social and online footprint creates an innovative mobile app persona so Martha can speak to him whenever she wants. AI’s algorithms predict and direct how Ash will respond and uses his own voice, which speaks conversationally as if he were still alive,

You can almost hear the hacker community planning the next innovation in social engineering.

Google Duplex Into the Breach

This summer, Google is rolling out its Duplex technology to a small trial set of business users and consumers. Google CEO Sundar Pichai demonstrated Duplex at the company’s annual developer conference in May. Pichai has Duplex call a hair salon to make an appointment, which it does in a natural, human voice, complete with “ums” and “uhs” and correct responses to a real human at the salon.

The Duplex neural network is trained on a database of phone conversations and takes advantage of Google’s continually improving automatic speech recognition (ASR). Although Google continues to emphasize the trial-like nature of Duplex, stressing that its use cases will be along a very narrow set of defined tasks, it is also built on the company’s TensorFlow, an open source framework for machine learning.

The Threat of Weaponized AI

Enterprise organizations have long been victimized by a variety of phishing attacks, some of them automated, some of them targeted, some of them using social engineering tactics, and far too many of them successful.

Business email compromise (BEC), typically deployed to trick users into acts like untraceable wire transfers, has generated more than $5 billion in financial losses between 2013 and 2016, according to IBM’s 2018 threat report. The FBI reports $1 billion in U.S. BEC-related losses in 2016 ($5 billion worldwide).

Targeted social engineering by phone has been a favorite hacker technique for decades, but it’s a time-consuming process outweighed only by its effectiveness. You can imagine, then, that the idea of automating phone-based social engineering using the latest AI innovations, like Google Duplex, is a tantalizing prospect for hackers. At the annual Black Hat conference in 2016, ZeroFox researchers John Seymour and Philip Tully presented “Weaponizing Data Science for Social Engineering,” in which they targeted Twitter as a source for a phone-based spearphishing scheme using AI.

Technology like Google’s Duplex begins to change the equation efficiency-wise as well. Double down on that efficiency with technology that could also mimic the voice of a CEO or CFO and BEC effectiveness goes up several notches.

Whether AI voice innovations are the future of social engineering is open to debate. Whether such schemes are making their way into the toolset of hackers is also unknown. But the mindset is established. Seymour told me that ZeroFox researchers are using Tacotron, an early Google text-to-speech synthesis model that was likely a precursor to Duplex, to defeat voice-based authentication systems.

Much has been made of CereProc, used to produce the speech—in his own voice—that President Kennedy was on his way to deliver in Dallas the day he was shot and killed for the film JFK Unsilenced. CereProc’s personal version, Cere Voice Me, costs about $650. A product called Lyrebird uses machine learning to take limited recordings of someone’s voice and read snippets of text as that person—with eerie accuracy. The idea, the company told TechCrunch last year, was to provide an API for third parties to start incorporating the innovative technology.

If the enterprise version of Ash may be on the way, how can we prepare?

Two-factor Authentication Can Make a Big Impact

Organizations would be wise to deploy two-factor authentication says Ryan Olson, vice president of threat intelligence for Palo Alto Networks. Google, for example, claims that its 85,000 employees haven’t reported a phishing attack in more than a year, thanks to the company’s deployment of physical security keys.

The Pros and Cons of the National ID Card

The effectiveness of systems that protect critical personal and corporate data is constantly challenged by systems that can emulate real people and real communication, Olson says. Simply put, we need better, stronger forms of authentication. Olson believes that this could take the form of a smart card, perhaps even on a national level for maximum effectiveness.

Olson notes that other countries—like Bangladesh, Jordan, and Belgium—have implemented national ID systems based on smart card chips for things like preventing voter (fraud).

Naturally in the U.S., this provokes concerns about privacy and encroachment on freedom, Olson notes. It raises questions about how companies, especially non-government entities, might take advantage of a single digital paper trail to be followed and targeted. In other words, this is more of a policy challenge than a technological one.

Tying a national ID card to convenient services, like the ability to make payments, replacing a passport, or using the card for voting, might make the privacy trade-off worthwhile for some people, Olson says.

Olson and other security experts I talked to believe that using technology like Google Duplex for social engineering and other phishing-style attacks is still a few years away, primarily because the technology to truly automate these experiences isn’t readily accessible. But the time to start planning for them is now, because you can be sure the attackers are already thinking about it.