Security leaders have already decided that AI agents belong in the SOC. In a Strike48 survey of 100 enterprise CISOs and mid-market security leaders, 84% said AI agents should be doing Tier 1 work, yet only 22% would fully automate it and just 36% had any agent running in production. Distrust of agent outputs was the top barrier, cited by 52%. Choosing a platform therefore comes down to authority, meaning which actions an agent may take alone and who answers when one goes wrong.
Security Digest interviews with practitioners show how buyers answer that question, and each starts from a different place. Mohit Bansal, Senior Manager of Security Engineering at Webflow, starts from the worst case. "Assuming that every agent that you've deployed is eventually compromised can help you design for survival," Bansal advises.
Mike Green, CISO at Availity, puts the verdict with the front line. "My analysts would have to be excited about it. The reality is, it's got to impress the folks who are actually on the front lines," Green says.
Aliyana Isom, founder of Sekena Solutions and former cybersecurity program lead at Nike, treats ownership as the first question. "When it comes to evaluation, I always ask who owns the new risk that the automation has introduced," Isom says. "If the answer is nobody, or there is finger-pointing, the organization has not reduced risk; it has obscured it."
The five criteria below draw on the full range of expert interviews Security Digest has run with security leaders, where the same tests keep coming up. To choose an AI SOC platform, check how much the agent may do on its own, whether every decision leaves a trail that can be replayed, how each agent's identity and access are controlled, how well the platform fits the SIEM already in place and how analysts score it in a 30-day trial. Contract terms and headcount plans then decide whether those answers still hold after signing.
Autonomy limits should follow reversibility
Agents should act alone only where a wrong call can be undone. Every other check builds on that limit, because an audit trail or an access policy means little until someone has decided what the agent may do unattended. Practitioners tend to draw the line by impact and reversibility, with model confidence as one input. Bansal puts it between the session and the production service. "If you have to kill a low-risk session or revoke a non-critical token, that is fine for the AI to handle," Bansal says. "But we cannot have AI just roam around and shut down services in production."
Someone has to draw that line before the alert arrives. Chris Patteson, director of cybersecurity, GRC at a large pharmaceutical manufacturing organization, says the business, security leadership and operations need to agree in advance on the triggers, the asset boundaries and the acceptable cost of an automated response, such as shutting down a network segment at a cost of a million dollars in lost production. Buyers should test whether a platform lets them set which actions an agent takes alone and which need approval, per workflow and per asset, and what the agent does when no one approves in time. Those settings should be changeable without the vendor's help.
Strike48's President and Co-founder,Tim Leehealey says the company gives customers the same guidance. "We tell teams to set autonomy workflow by workflow, because a wrong call on a phishing alert and a wrong call on a production server do not cost the same. Start where the impact is low, run the agent beside your analysts, and widen its limits once the results hold up."
A decision trail has to be replayable
Limits only count if the agent can be shown to have stayed inside them. When an analyst doubts a verdict, someone has to be able to reconstruct what the agent did, why, on what data and what triggered it. Bansal sets the logging bar by consequence. "Every write, every irreversible action needs to be logged," he says. Entries should carry a timestamp and trace information showing where the command came from, which matters most when one agent may have triggered another. A parent trace ID on each command lets a team tell whether an agent started a malicious action itself or received instructions through a compromised component.
Reporting deadlines raise the stakes, because a business that cannot explain what an agent did then has a compliance problem. Buyers should check that every tool call is logged, that logs export to their SIEM and that retention matches their audit rules. A useful trial test is to pick alerts at random and have the vendor reconstruct the decision.
Strike48's President and Co-founder,Tim Leehealey frames the same need from the analyst's side. "If an analyst cannot see why the agent closed an alert, they will redo the work, and the time savings disappear. Every action an agent takes should come with the evidence behind it, in a form an analyst can check in minutes."
Each agent needs its own identity and permissions
A trail shows what an agent did, but identity and access settings decide what it could have done, so access control, agent identity and kill switches belong in every evaluation. Permissions can also compound in ways buyers miss. "If Agent A can communicate with Agent B, and Agent B has write access, that inherently means Agent A also effectively holds that write access—even if it was not directly granted," Bansal explains. He recommends first knowing which agents operate in the environment, what dependencies and permissions they hold, and which systems those permissions expose.
In practice, that means each agent should run under its own least-privilege identity, the product should enforce which roles can create, edit and approve agents, and the kill switch should have a named owner and a defined scope. Role-based access control covers people as well as agents, so buyers should know who on their own team can create, change or approve agent actions. Multi-tenant governance matters mainly to MSSPs and to companies with separate business units, where one tenant's data and agent settings must stay apart from another's. Buyers outside those groups can rank it lower.
The platform has to fit the SIEM already in place
A well-governed agent still has to work inside the stack a team already runs. A product that brings a new console, or needs data copied into a new store, creates work for the people it was bought to relieve. Benjamin Glover, a manager of information security, applies a simple filter. "The first thing I look for in any tool is whether it's going to create another management plane," he says. "Is this another portal my team has to check every day? If so, that rules it out automatically."
Fit also depends on data access. In Strike48's survey, 84% of respondents said their current tools cannot reach all of their log data for investigations. And 65% had at least one investigation stall in the past 12 months because the data they needed sat in a system their tools could not access. An agent working from a partial picture of the environment tends to extrapolate, and in security operations the line between extrapolation and hallucination can disappear after the fact. Buyers should ask whether the platform works alongside the SIEM or requires replacing it, whether it searches data where it lives or requires a copy, and which connectors exist today and whether they are two-way.
A 30-day trial should end with analysts scoring the platform
Unlike a demo, which shows a platform on a good day, a 30-day trial beside the current process shows what it does with a team's own alerts and data across a full month of normal alert volume. Results can then be scored against the analysts' own verdicts on triage accuracy, false positives closed correctly, escalation accuracy, time to resolution, MITRE ATT&CK technique coverage and the share of an analyst's day spent on routine work.
Time to resolution depends on people as well as the agent. Agents can surface recommendations faster than people can review them, so approvals queue up and human review becomes the bottleneck. A trial should measure how long approvals wait alongside how fast the agent works. Buyers should also agree in advance on who defines the pass criteria and ask the vendor to share the full trial data, including the agent's misses.
A good trial still leaves three questions the product cannot answer. The first is who owns the risk the automation creates, which returns to the question Isom asks of every implementation. Someone has to be accountable if an agent action causes an outage, and the answer should be named before signing.
The second is what the buyer will be locked into. Brad Gorka, CISO at Veritiv, keeps contracts to two years at most and settles his exit strategy before he signs. He also asks vendors to commit in writing that features promised at signing stay available and that price increases carry a ceiling. One large vendor that called an AI feature included refused to put it in the agreement, though his team did win a cap on increases from one contract to the next. Buyers should ask for AI features and price caps in the agreement, along with the exit path and data export terms.
The third is what happens to the team. Isom warns that treating automation as a quick way to cut headcount can weaken security posture if the remaining team lacks the skills to oversee the systems. The staffing plan should be settled before signing.
How Strike48 approaches these checks
Strike48 builds its platform around several of these criteria. The company says teams choose which actions agents take on their own and which need human approval, and that every agent action is traceable and auditable, showing what an agent did and why. The platform searches data in place across more than 300 systems, including SIEM, observability, ticketing and identity tools, connected by API or MCP. Pre-built agent packages ship with approval gates and audit trails, and teams can customize them or build their own.
Those are claims to verify in a trial, as with any vendor. Ask to see the approval settings per workflow, the trail for one of your own alerts and the connector list for your stack. Strike48 also describes multi-tenant isolation, with per-tenant namespaces, quotas, role-based access control and separate secrets, so MSSPs and businesses with separate units should test how that holds up across tenants.
Tim Leehealey invites that scrutiny. "Customers set the approval gates, and every action an agent takes leaves a record they can inspect. We'd rather buyers test both on their own alerts than take our word for it."
Frequently asked questions
Is an AI SOC platform just SOAR with AI?
That depends on what the platform does with each alert. The test is whether the agent chooses its own steps for each alert and records why, or only runs fixed playbooks faster. Ask the vendor to show it on one of your alerts.
What should I ask an AI SOC vendor?
Six questions cover most of it: what the agent may do alone and who sets the limit, whether every decision can be replayed, how each agent's identity and permissions are controlled, which SIEM and tools it works with, how it performs on your alerts in a trial and what the contract locks in.
How do I audit an AI SOC agent's decisions?
Require a record of every action showing what the agent did, why, on what data and what triggered it. Log writes and irreversible actions with timestamps and trace information, and check that the logs export to your SIEM. In the trial, pick alerts at random and ask the vendor to reconstruct the decision.
How long should an AI SOC proof of value run?
A 30-day trial run in parallel with your current process is a practical starting point. It gives the platform a full month of your normal alert volume. Score it against your analysts' own verdicts and track how long approvals wait.
Will an AI SOC platform replace analysts?
Not on the evidence so far. Only 22% of leaders in Strike48's survey would fully automate Tier 1 work, and practitioners keep people on irreversible actions. What changes is the work. Agents take on routine triage, while analysts review agent decisions, handle the cases the agent can't and set the approval limits.
Do I need to replace my SIEM to use an AI SOC platform?
Not necessarily. Platforms differ, so ask whether the platform searches data where it lives, which connectors it has and whether it adds another console for the team to check.
