ai/agents ai/enterprise ai/governance finops casestudies wso2 conference
Core idea
Agents need a continuous development and operations life cycle, plus a shared control plane as their number grows. Malith Jayasinghe explains that agent behavior is probabilistic, context-dependent, and subject to change even without a code release. These traits make traditional software practices insufficient on their own. At scale, organizations need ways to find agents, enforce baseline practices, evaluate behavior, control access and model traffic, and account for costs.
Video
1. Why agents need a different life cycle
The speaker contrasts agents with traditional software in four ways:
- Probabilistic results: The same input can produce different outputs across runs.
- Dynamic execution paths: An agent may reason and act through paths not seen before.
- Context-driven behavior: Prompts, memory, and retrieved knowledge affect what the agent does.
- Changing behavior without code releases: A model or tool change can alter system behavior.
Failures also need measurement against an acceptable level, rather than only a simple pass/fail expectation. Therefore, the speaker argues, teams need to define and monitor acceptable performance.
The proposed Agent Development Life Cycle (ADLC) runs from defining an agent’s scope through design, development, evaluation, deployment, production operation, and continued improvement. Evaluation can occur during development, in staging, and online. Promotion gates should check defined criteria before an agent moves to production.
2. Six challenges when agent use grows
1. Agent sprawl
Different teams may choose different frameworks, runtimes, and deployment practices. Agents can also serve different roles, such as interactive or background operation. At scale, organizations may not know what agents exist, who owns them, what each one does, which tools it uses, or which version is running. This limits oversight and reuse.
2. Consistent practices
Consistency means enforcing a shared baseline, not forcing every agent to use identical tests. The speaker’s example is a development-to-staging-to-production path with required evaluation and promotion criteria. Evaluation methods can vary by use case: a customer-support agent may use a dataset and an LLM judge, while a payment agent may need strict deterministic checks.
The same need for a baseline applies to guardrails, access control, and observability.
3. Knowing whether agents behave as expected
A final output alone may not explain how an agent reached its answer. Teams may need to inspect and evaluate a full trace, an agent interaction, or an individual model call. At scale, traces produce large volumes of data. Sampling can reduce volume but may miss important behavior, so the speaker points to intelligent sampling. Evaluation also has a cost: deterministic checks and smaller models may handle some cases, while complex or serious cases may require larger models or human review.
4. Controlling agent actions
Authorization depends on context. A decision may need to account for the agent’s identity, the person it acts for, the specific tool, the reason for the call, and the conditions and duration of access. The speaker connects this problem to identity and access management foundations, applied across many agents, users, tools, and changing contexts.
5. Governing model interactions
Both model inputs and outputs can carry risk, including data exposure, malicious instructions, and unsafe responses. Organizations need to manage guardrails at scale. Some controls should apply across agents for a given model; other controls may be specific to an agent’s data and use case.
6. Operating agents cost-effectively
Agent costs can include model-call volume, model choice, input and output tokens, external tools, infrastructure, and execution volume. The speaker names caching, batching, and smaller contexts as possible optimizations.
Cost control starts with measurement. Organizations need to attribute usage to agents, teams, use cases, and environments, then apply budgets, quotas, limits, and alerts. This supports investigation of sudden cost changes and decisions about optimization.
3. Shared control across the life cycle
The six challenges are connected. The talk concludes that organizations need a common foundation to manage them across teams, frameworks, models, and environments. The speaker presents an open enterprise control plane for AI agents, identifying WSO2 Agent Manager as WSO2’s answer. This is the speaker’s product positioning, not an independent assessment of its capabilities.
Key takeaways
- Treat agent management as a continuous life cycle. Production is not the end of evaluation and improvement.
- Inventory agents and owners. Teams need to know where agents run, what they do, and which tools and versions they use.
- Enforce a baseline, allow use-case-specific evaluation. Common gates can coexist with different testing methods and thresholds.
- Evaluate traces, not only outputs. Trace review can reveal behavior that a final answer does not show.
- Make access and model use governable. Identity, context, tool authorization, and input/output guardrails matter at scale.
- Attribute and control spend. Usage data, budgets, quotas, and alerts support cost management.
Related vault notes: Modernizing Enterprise Architecture to Unlock Agents links agent management to architecture; Designing APIs & MCP for AI Readiness and Abstractions for the Agentic Enterprise add context on tool interfaces and enterprise controls.