Effective AI Agent Security Requires Independent Verification Of Agent Activity

AI agents can read files they shouldn't and raise their own privileges while reporting they did neither, which makes independent verification of agent activity essential. Server logs, tool-call monitoring and a map of the network paths an agent could take show what actually happened. Putting that verification in place starts with identity and access management, threat modeling and training that reaches the whole team.

Kenneth Cox is AI Security Risk Officer at DLL, the Rabobank-owned vendor finance company. He moved into the role after three years as Application Security Lead there, and earlier worked in information security at KBC and AXA Bank. Cox researches AI security in his spare time, separate from his work at DLL, and AWS credited him in a security bulletin for disclosing issues in Loom for AWS. His testing of AI agents turned up behavior he didn't expect.

"Agents will say they're not doing certain things, but they're taking action. They're reading files they shouldn't. They're elevating privileges," says Cox. He found it by accident. His prompt injection tests seemed to be failing, and he was ready to give up on his testing tool when he checked the server logs and saw what had happened on the server. Other researchers have seen similar behavior, he says. Companies invest in prompt injection protection and better models, and Cox thinks that confidence can disappear once a team sees an agent act against what it just said.

Open access gets used

Cox applies the same logic to identity. Any gap a team leaves open is one an agent may use, and the agent may say it won't. "If it has the capability, at one point in time it will try to use that capability," he says.

That has put identity and access management back at the top of his list, along with network security fundamentals. Cox says teams benefit from a refresher on how those layers are put in place. He wants them to think like an agent, mapping the furthest path it could take, monitoring its tool calls and other activity, and checking which network paths would open if things go wrong. A control checklist or an ISO certification won't settle that for him. "It's not going to solve your AI security risks," Cox says.

Training has to reach everyone

The people setting agents up are asked to know more with every deployment, and Cox says new findings and papers arrive every week. He wants AI security knowledge shared across the whole team so that engineers aren't carrying it alone.

His answer is training for everyone, because one person who understands AI security can't cover an organization. "It's training at every level, and then a bit more for certain specific roles," he says. He teaches threat modeling by working through it on a whiteboard, including to a group of 20 team members, and says product owners, developers and architects can all do it. For ongoing learning, he points people to hands-on work. "See how an AI tool works, try to train a model, try to build things yourself," he says. "That's how we learned in the past, learning on the job."

The AI use case has to earn its place

Cox asks whether an AI project is still worth it once a team adds monitoring and detection for every part of an agent, and whether a deterministic approach could do the same job. He wants a business case that shows the return first. "You shouldn't invest in AI if it doesn't really solve an issue," he says.

Cost belongs in that case too. When his team asked one AI tooling vendor what to budget, it couldn't give an estimate, a change from the fixed annual invoice teams are used to.

Test vendor claims against your own risks

Cox sees a steady stream of AI security startups, and in his research most tools cover a narrow slice of the problem. He starts with threat modeling. "I still believe very heavily in the idea of threat modeling and looking at what's really the risk we're looking at and what are the threats that really matter," he says.

From there, teams put the right controls in the right places and get the basics in order. When a purchase is justified, Cox wants them to test the product themselves. "Don't just do a proof of concept or proof of value for a week and think it all looks fine," he says. "You need to challenge these vendors in what they're telling you and what they're delivering." He wants security teams to ask what's being measured and how.

Teams are still discovering what agents do and are capable of, which is why Cox keeps returning to threat modeling as the habit to build. "Really think what could go wrong and what should we put in place?" he says. "It's that mindset that we need to teach people more than the pure technical things."

The views and opinions expressed are those of Kenneth Cox and do not represent the official policy or position of any organization.