AI ROI at scale: the model, architecture and adherence problem

Don't use the most expensive model for every decision

One of the easiest mistakes to make with AI is to select a highly capable model and then use it for everything. It works, the quality is good, the implementation is relatively simple, so the model becomes the default. But an AI process rarely consists of one type of task.

A customer service agent might need to;

Those tasks don't necessarily require the same level of reasoning. Some may be deterministic, some may be handled by a smaller model, some may require a more capable model, and some may not require an AI model at all. This is why model selection needs to become part of the architecture rather than an afterthought. AWS, for example, recommends mapping agent tasks to the smallest model that meets the required quality bar, with escalation to a more capable model when necessary.

The objective isn't to use the cheapest model, it is to use the right model for the job.

The economics of the agent need to be understood at task level

This becomes even more important with agents, a traditional AI interaction might involve one model call, an agentic workflow can involve multiple calls, tool use, planning, validation, retries and escalation.

That means the cost of an agent isn't simply:

Cost of the model × number of users

It can be closer to:

Cost per task × number of steps × number of executions

And if the agent starts retrying, calling additional tools or escalating unnecessarily, costs can compound, that is why measuring tokens alone isn't enough.

The more useful measure is:

Cost per successfully completed outcome.

A cheaper model that fails 20% of the time and requires expensive retries may be more expensive than a larger model that completes the task successfully the first time.

Recent research into enterprise AI cost optimisation is increasingly looking at this distinction between token cost and the cost of successfully completing the task.

Architecture before scale

This leads to an important design principle:

Don't wait for AI costs to become a problem before designing the cost architecture.

Before an agent goes into production, organisations should understand: What is the task? What level of reasoning does it require? Which model should perform it? What is the maximum number of attempts? When should it escalate? What tools can it access? What data can it access? What actions can it take? What happens when it fails? What is the expected business outcome?

These aren't separate technical questions, together, they define how the AI worker operates.

This is where adherence becomes important

We've spent a lot of time thinking about adherence in the context of AI agents, the concept comes from something familiar in traditional operations. In a contact centre, for example, it isn't enough to know that an employee handled calls, organisations can define an operating process and measure whether the employee adhered to it. The same concept becomes increasingly relevant when the worker is an AI agent.

An agent may have been designed to:

That is the agent's intended operating process. But what actually happened? Did it follow the process? Did it access information it wasn't supposed to? Did it skip a validation? Did it make an unnecessary model call? Did it escalate when it should have? Did it continue trying after it had reached a defined threshold? Did it achieve the intended outcome?

This is AI adherence.

Adherence isn't just about compliance

This is an important distinction. adherence shouldn't simply mean:

"Did the agent follow the policy?"

It should also mean:

"Did the agent operate the way we designed it to operate?"

That includes the economics. Imagine an agent is designed to resolve a customer request in four steps, instead, it takes eleven. It eventually produces the right answer, from a traditional AI quality measurement, that may look successful. From an operational perspective, something has gone wrong, the agent has achieved the outcome, but inefficiently. The additional model calls, context and tool usage have increased the cost of that outcome.

So we need to measure both;

Outcome adherence

and

Operational adherence.

The model should be part of the operating design

This changes the way we think about AI architecture.

Rather than:

we start thinking about:

The model becomes one component of a larger operating system, and the choice of model can change depending on the task.

For example:

The exact architecture will vary by organisation and use case. The important point is that model selection should follow the business process, not the other way around.

TraphicLights: making the operating model visible

This is where TraphicLights fits into the architecture.

The challenge isn't simply identifying which models an organisation is using.

It is understanding the relationship between: Agent, Owner, Process, Model, Data, Decision, Action, Cost, Outcome and Adherence.

That creates a much more complete picture of AI operations, an organisation can start to ask:

Which agents are using expensive models? Which processes generate the most AI spend? Which agents are making unnecessary calls? Where are retries occurring? Which agents are escalating? Which workflows are operating outside their intended process? Which model is actually delivering the best cost-to-outcome ratio?

And ultimately:

Which AI processes are creating measurable business value?

Scaling AI should mean scaling value, not just usage

This brings us back to ROI, the objective shouldn't be to minimise AI costs at all costs, nor should it be to maximise AI adoption. The objective is to maximise business value per unit of AI investment.

That requires balancing: Capability, Cost, Quality, Risk, Speed and Outcome. An organisation might deliberately use a more expensive model for a high value decision because the additional capability produces a materially better outcome, that can be good economics. Conversely, using a frontier model for millions of routine classifications simply because it is the default may create unnecessary cost.

The answer isn't "always use the smaller model."

It is:

Use the model that makes economic and operational sense for the task.

From AI governance to AI economics

This is why we think AI governance is increasingly becoming an AI operations problem. Once AI agents are making decisions and taking actions at scale, organisations need to understand not only whether they are compliant, but whether they are operating effectively.

Governance tells us what an agent should do. Adherence tells us what it actually did.

Operational measurement tells us what that behaviour cost. Outcome measurement tells us what the organisation got back.

Put those together and we can start to calculate something much more meaningful:

AI ROI.

The Traphiclights architecture for AI ROI

We see the journey as:

1. Define the outcome

What business result are we trying to achieve?

2. Design the process

What should the AI actually do?

3. Assign the right model

What is the minimum capability required for each task?

4. Define adherence

What behaviour is expected from the agent?

5. Set operating boundaries

How many calls, retries, tools and actions are permitted?

6. Measure actual behaviour

What did the agent really do?

7. Measure cost

What did it cost to complete the task?

8. Measure the outcome

What business value was created?

9. Continuously optimise

Can the process, model or architecture be changed to improve the cost to outcome ratio?

That is a very different approach to simply looking at an AI invoice at the end of the month.

The next AI optimisation problem

As AI becomes embedded in business operations, the question will increasingly move from:

"How much does AI cost?"

to:

"What does it cost us to achieve this outcome with AI?"

And then:

"Can we achieve the same outcome with a better architecture?"

That is where model routing, agent design, adherence and governance become connected. The organisations that solve this well won't necessarily be the ones using the most powerful models, they will be the ones that understand when to use which model, how the agent should operate, what it actually does, what it costs and what outcome it creates.

That is the foundation for scaling AI economically.

And ultimately, that is the difference between deploying AI and operating AI as part of the business.