Researchers with security firm Darktrace said they have observed a huge spike in the number of novel social engineering attacks via email since the start of 2023—a trend that could be linked to the relative ease of writing convincing phishing and scam emails with generative AI.
Darktrace chief product officer Max Heinemeyer told IT Brew that across the company’s over 3,000 customers, it had detected a 135% increase in malicious cyber campaigns via email that showed significant linguistic deviation “in syntax, in semantics, in grammar and sentence structure.” He said while the firm could not demonstrate causation or correlation and doesn’t want to fuel AI hype, the time period lines up with widespread commercial availability of tools like OpenAI’s ChatGPT.
“That means less spelling mistakes, longer sentences, better punctuation, more interesting topics, trying to incite people to do something,” Heinemeyer said, adding the attacks were clearly more sophisticated. “There is much more variety in the social engineering we’re seeing come in.”
Darktrace continually compares incoming phishing attacks to prior ones on “hundreds of metrics,” Heinemeyer told IT Brew. The recent data points, he said, “point in the direction of changing techniques, tools, and procedures, at least in the email threat landscape,” specifically that generative AI may be lowering the barrier to tailoring attacks towards specific targets.
There’s already evidence that recent generative AI products are finding widespread use in scams and fraud. For example, the Washington Post reported cybercriminals are already using voice cloning to try and trick family members of a target into wiring cash for bail or other emergency legal fees.
Top insights for IT pros
From cybersecurity and big data to cloud computing, IT Brew covers the latest trends shaping business tech in our 4x weekly newsletter, virtual events with industry experts, and digital guides.
Darktrace also commissioned a Censuswide poll of 6,711 employees across the US, Europe, and Australia in March 2023. It found around 70% of respondents said they had noticed a rise in the frequency of scam emails and texts in the past six months. One troubling finding was that the qualities that employees said they viewed as red flags for malicious messages may be ones which generative AI is particularly adept at improving on:
- 68% of respondents said they found invitations to click a link or open attachment as a factor that raises their suspicion.
- 61% said they were on the lookout for unknown senders or unexpected content.
- 61% are suspicious of poor use of spelling and grammar.
Generative AI may, for example, be able to better frame unusual requests within a context that makes them seem less unexpected, or better pose as an individual belonging to a specific organization or industry. Existing models are also quite good at grammar.
“The most dangerous phishing emails are the ones that are bespoke and well-crafted and tailored to the recipient,” Heinemeyer told IT Brew. “Doing that at scale needs an attacker to spend a lot of time—they need to research the victims, understand what they do.”
“Instead of sitting down and doing all of that research yourself, and then coming up with topics, you could just scrape a victim’s social media profile, or take a Newswire from a company, and then ask a generative AI model to create a believable email based off that, that’s angled towards inciting clicking on a link,” he added.—TM