From Token Counts to Token Value

In this workshop, we’re going to cover how agentic applications consume tokens and how we can validate that those tokens are effectively used.

Our goal is not simply to minimize tokens. The cheapest agent that produces the wrong answer is not efficient. Neither is a highly accurate agent whose unnecessary loops, oversized context, and expensive model choices make it impossible to scale.

The question we will keep asking is: what outcome did we receive for the tokens we spent?

We will use a number of tools, both reactive and proactive. We will investigate the data from a high level, all the way down to each granular trace records, reviewing every LLM invocation, chunk retrieval, and tool call.

LLM provider bills by themselves do not explain:

• Which users, agents, models, and workflow steps consume the tokens?
• Are expensive requests better than inexpensive requests?
• Does a prompt, model, retrieval, or tool change improve the ratio?

If your bill doubled for AI this month, what would you check to make sure it was justified?