ai/llm ai/governance ai/costoptimization apimanagement casestudies wso2 conference
Video
Why API request counts do not explain AI cost
Anusha Jayasundara argues that AI APIs do not have the same cost profile as conventional REST APIs. A request can trigger model inference, retrieval, guardrails, and multiple agent steps. The cost also depends on the size of the input and output, not only on how many requests arrive. A short question may use little, while a request that produces a long answer or sends extensive context may use much more.
The speaker uses a $0.37 per-request example to show how a seemingly small unit cost can grow at high request rates. This is an illustration, not a reported average cost.
Subscription plans based only on request counts can therefore misrepresent provider costs. Jayasundara describes Cursor changing from request-based to token-based pricing after its costs became unsustainable, and says the change caused a negative customer response. The example illustrates why pricing and consumption controls need to reflect actual usage.
What to measure and attribute
| Cost driver | Why it matters |
|---|---|
| Input and output tokens | Token volume is more directly related to consumption than request count. |
| Context size | Large prompts and retrieved context increase input tokens. |
| Model selection | A premium model may be unnecessary for a simple task. |
| Agent steps and retries | One user task can create many billable model calls. |
| Cached tokens | Some providers expose additional usage measures, such as cached input tokens. |
Usage data also needs business context. The speaker names seven useful attribution dimensions: organization, application, user, API, model, workflow, and outcome. These connect technical activity to the teams, workloads, and business results that own it.
Follow the request end to end
Jayasundara describes four stages for building an attribution record:
- Identity: Record organization or tenant, application, API, subscription, user or agent identity, and cost center where available.
- Execution: Track the trace ID, selected model, provider, and region.
- Consumption: Capture input, output, and cached token counts.
- Outcome: Record success, latency, result, and an outcome ID.
A trace ID that spans these stages makes it possible to investigate how a request was executed and what it consumed.
Reconcile estimates and provider invoices
The gateway can estimate cost per request when it has usage data and current pricing rules. Aggregate those estimates by model, provider, team, or time period, then compare them with the provider’s invoice. The speaker treats the invoice as the final source of truth. Differences can point to outdated pricing policies or other gaps in the estimate.
Analytics supports this feedback loop. It aggregates and meters usage, supports alerts, and helps teams identify patterns before they set or revise gateway controls. Gateway policies can then limit, route, cache, or block requests based on observed use.
Turn evidence into controls
The talk recommends diagnosing cost increases rather than relying on request volume alone. In one example, analytics reveals that a 60% increase in LLM spend came from a single internal application, where retrieval was sending more context. A team could respond by limiting the context at the gateway, then monitor whether the change improves usage.
Controls discussed include:
- Near-term: semantic caching, output-token caps, retry budgets, and routing work to an appropriate model.
- Longer-term: token budgets by team or workload, and prompts designed to use caching effectively.
- Governance: guardrails for permitted use, and alerts or limits based on usage patterns.
These controls have trade-offs. The speaker advises naming those trade-offs before shipping a policy. Analytics and gateway enforcement must work together: data informs policy, and the gateway applies it.
Key takeaways
- A request count is not a cost measure. Track token use, model choice, context, retries, and other billable work.
- Attribute each call. Capture identity, execution, consumption, and outcome details so teams can find who or what drives spend.
- Use analytics to diagnose before enforcing. Find the application, workflow, or context pattern behind a cost increase.
- Reconcile gateway estimates with provider invoices. Pricing and usage records can change, so estimates need validation.
- Treat attribution as a business capability. The speaker frames it as a foundation for governance and possible monetization, not only expense reporting.
Related vault notes: LLMs and AI Agents explains context-token demand; Managing the Agent Lifecycle at Scale covers agent-level FinOps controls; Designing APIs & MCP for AI Readiness connects usage costs to API and tool design.