Abstract
The rapid adoption of large language models (LLMs) has fueled speculation that cybercriminals may utilize LLMs to improve and automate their attacks. However, so far, the security community has had only anecdotal evidence of attackers using LLMs, lacking large-scale data on the extent of real-world malicious LLM usage. In this joint work between academic researchers and Barracuda Networks, we present the first large-scale study measuring AI-generated attacks in-the-wild. In particular, we focus on the use of LLMs by attackers to craft the text of malicious emails by analyzing a corpus of hundreds of thousands of real-world malicious emails detected by Barracuda. The key challenge in this analysis is determining ground truth: we cannot know for certain whether an email is LLM or human-generated. To overcome this challenge, we observe that, prior to the launch of ChatGPT, email text was almost certainly not LLM-generated. Armed with this insight, we run three state-of-the-art LLM detection methods on our corpus and calibrate them against pre-ChatGPT emails, as well as against a diverse set of LLM-generated emails we create ourselves. Since the launch of ChatGPT, all three detection methods indicate that attackers have steadily increased their use of LLMs to generate emails, especially for spam. Using our most precise AI-detection method, we conservatively estimate that at least ∼51% of spam emails and ∼14% of business email compromise attacks in our dataset are generated using LLMs, as of April 2025. Finally, analyzing the text of LLM-generated emails, we find evidence that attackers use LLMs to “polish” their emails and to generate multiple versions of the same email message.
Coverage
- Forbes AI Is Behind 50% of Spam — And Now It’s Hacking Your Accounts
- TechRepublic AI Now Creates 51% of Spam: Two Key Reasons Attackers Use This Approach
- Infosecurity Magazine AI Now Generates Majority of Spam and Malicious Emails
- The Register AI Improves Spam
- Columbia Engineering AI Now Powers Over Half of Spam Emails, Columbia Engineering Research Finds
- Columbia Magazine The Deepfake Scam Era Is Upon Us. Here’s How to Get Ready.
- Barracuda Half the Spam in Your Inbox Is Generated by AI — Its Use in Advanced Attacks Is at an Earlier Stage
- TechRadar Pro Major Spam Email Warning — AI Now Generates Almost All of Your Junk Mail
- Cybernews Scammers Now Using AI for A/B Testing
- Security Boulevard Analysis Surfaces Increased Usage of LLMs to Craft BEC Attacks
- Channel Insider AI Now Powers Over Half of Global Spam Emails, Research Finds
- Cloudflare Attackers Go Phishing in New Ponds
- DIGIT Over Half of All Spam Emails Are Now AI-Generated
- CybersecAsia Fraudsters and Cybercriminals Tap AI for More Sophisticated Spam and BEC Attacks
- SecurityBrief Australia Over Half of Spam Emails Now Generated by AI, New Study Finds
- ANSA Il 51% delle email indesiderate è generato dall'IA