ServiceNow's EnterpriseOps-Gym wants to stress-test AI agents before they touch real work

The gap between what AI agents can do in a lab and what they can reliably do inside a company has always been uncomfortably wide. Chatbots that ace standardized benchmarks routinely stumble when handed a real procurement workflow or a multi-step IT service request. ServiceNow Research, working alongside Mila, thinks it has identified why: the benchmarks themselves are broken.

The team has introduced EnterpriseOps-Gym, a high-fidelity evaluation environment designed specifically to test whether large language models can function as autonomous agents inside the messy, rule-bound reality of enterprise operations. The project targets three properties that conventional benchmarks almost entirely ignore: long-horizon planning, persistent state changes, and strict access protocols. Each of those properties sounds technical, but together they describe something very human, which is the experience of navigating a large organization where one wrong click has consequences that outlast the conversation.

Most AI benchmarks are, at their core, single-turn affairs. A model reads a prompt, produces an answer, and gets scored. Even the more sophisticated multi-step evaluations tend to reset the environment between tasks, meaning the agent never has to live with the downstream consequences of an earlier mistake. That design choice made sense when researchers were primarily testing reasoning ability. It makes much less sense when the goal is to deploy an agent that will autonomously open tickets, modify database records, or route approvals through an HR system.

Enterprise software environments are stateful. An action taken in step three of a workflow changes what is possible in step seven. Access controls mean that certain information is simply unavailable to certain roles, and a well-designed agent needs to recognize that boundary rather than hallucinate its way past it. These are not exotic edge cases. They are the baseline conditions of professional software use, and until EnterpriseOps-Gym, there was no rigorous way to measure whether an AI agent could handle them.

The timing of this research reflects a broader pressure building across the industry. Enterprises are being pitched aggressively on agentic AI, the idea that LLMs can move beyond answering questions and begin completing tasks autonomously. ServiceNow itself has commercial stakes in that vision. But the credibility of the entire agentic AI category depends on being able to demonstrate, with reproducible evidence, that these systems do not silently corrupt workflows or violate access boundaries when left unsupervised. EnterpriseOps-Gym is partly a scientific contribution and partly an infrastructure investment in that credibility.

There is a systems-level consequence here that deserves more attention than it typically receives. When AI agents operate in persistent-state environments, errors do not stay local. A misrouted approval in a procurement system can delay a supplier payment, which affects a vendor relationship, which surfaces weeks later as a supply chain gap. The causal chain is long and the feedback is slow, which is precisely the kind of environment where human oversight tends to degrade over time. People stop checking the agent's work because it usually looks fine, right up until it does not.

Benchmarks like EnterpriseOps-Gym matter because they create the possibility of catching those failure modes before deployment rather than after. If a model consistently mishandles access-restricted data in a simulation, that is a signal worth acting on. If it handles simple tasks flawlessly but degrades on step twelve of a fifteen-step workflow, that degradation curve is something a procurement team needs to know about before they hand the agent the keys to their supplier portal.

The collaboration with Mila, one of the world's leading academic AI research institutes, also signals something about where serious AI safety and evaluation work is gravitating. It is moving closer to the operational layer, closer to the specific software systems that organizations actually run on, and further from the abstract reasoning puzzles that dominated the previous generation of benchmarks.

What EnterpriseOps-Gym cannot yet answer is the question of organizational adaptation. Even a perfectly calibrated benchmark measures agent behavior in a controlled simulation. Real enterprises will modify their workflows, layer on legacy systems, and introduce human interruptions that no gym environment fully anticipates. The more interesting test, and the one that will take years to run, is whether the organizations deploying these agents build the institutional habits needed to monitor them over time, or whether the appearance of competence quietly substitutes for the reality of it.