Abnormal Security wants to make sure that email from human resources is actually from a human. To that end, the email protection provider must incorporate new suspicion-seeking signals into its machine-learning detection technology.
That includes questioning quirky behavior by the aforementioned HR department. Dan Shiebler, head of machine learning at Abnormal, gets the company’s model to spot behavior characteristics of the tactic—maybe a common word choice featured in messages, a high frequency of sends, a strange header, or a trait in the sender’s email infrastructure.
Just incorporating one signal into the company’s machine-learning systems requires 12 manual, “error-prone” coding steps, or file changes, Shiebler said. A new signal means, at the very least, a new data-schema column in the data table, and potentially code to extract it, and a test to demonstrate the extraction worked.
Shiebler recently deployed an agent to automate those manual tasks, which employees can prompt to create code that incorporates new signals into their detection models and performs the dozen steps autonomously.
“Before…every time I’d think of a signal, it was like, ‘Oh my god, I gotta make all these changes. I have to clear my afternoon to implement this,’” Shiebler told us.
Any ideas? Weeks ago, Abnormal conducted an internal “hackathon” competition for its engineers to find ways for the company to use Cursor—a platform that outputs code based on natural-language prompts. Following the brainstorm, Shiebler went to work creating an agent for the signal-adding effort.
Shiebler fed Cursor commands that got the agent to understand its job—These are the 12 files that need to be changed when an attribute is added. These are approximately the kinds of changes you’ll need to make for each file, to paraphrase a training effort that usually takes more text.
Top insights for IT pros
From cybersecurity and big data to cloud computing, IT Brew covers the latest trends shaping business tech in our 4x weekly newsletter, virtual events with industry experts, and digital guides.
The prompting began with a little Caps Lock: Don’t be an agent that tries to do too many smart things. Just follow these steps EXACTLY, Shiebler wrote.
Shiebler then wrote prompts for subsequent processes like updating schemas (“This should be a very small change…”); writing tests (“Do not try to run this test, just do your best shot of writing something basic…”); and implementing extractors.
A “suite” of automated and human testers double-check that the agent incorporated the feature correctly, according to Shiebler. “The output of this is like a pull request and needs to be reviewed,” he said.
Who else is in? Deloitte’s 2024 Q4 survey found that 26% of 2,733 global business leaders said their organizations were “already exploring autonomous agent development to a large or very large extent.”
Getting the prompts and outputs right took a few hours and some trial and error, Shiebler said, but the process works so far.
“I can just go to a natural-language interface and describe what I’m trying to do, and an agent on the back-end will make all of the changes that are needed, test all of those changes, launch into the system, and it’ll be automatically incorporated into our models,” Shiebler told us. “That just dramatically cuts down the amount of time required in order to make improvements to the machine-learning models that detect cyberattacks.”
What can he do with that new time?
“Think of more of these! Think of more signals,” he said.