Postman says tool sprawl, not model quality, was the hard part of shipping Agent Mode

Postman and AWS have published a joint account of how Postman built Agent Mode, the company's AI-native entry point for API testing, documentation, discovery and implementation. The headline finding is not about model quality: according to the post, the Postman team expected prompt design and model choice to be the hardest problems, and instead ran into structural issues inside an 11-year-old product — tool catalogs that grew past what a model could reliably choose from, client APIs welded to interface state, and context windows filled with data shaped for rendering rather than reasoning. The write-up is a vendor post, so it describes what worked for Postman rather than offering independent evidence, but the failure modes it names are the ones most teams hit second.

The 40-tool cliff

The most concrete number in the post is an accuracy threshold. Postman says that in its own testing, tool-selection errors increased once the agent could see more than roughly 40 tools. The failure modes were familiar: calling tools that did not exist, passing wrong arguments even when schemas were valid, and picking tools that looked semantically plausible but were wrong for the situation. Larger and newer models reduced the behavior, Postman says, but did not eliminate it.

That threshold is not presented as a general law, and it is worth treating it as a property of Postman's specific tool naming, descriptions and domain rather than a universal constant. The methodology behind the number — what counted as an error, how many trials, which models — was not disclosed. Still, the directional claim matters: past some catalog size, adding tools makes an agent worse, not better.

Postman's response is dynamic scoping. A root agent queries a vector database of tool embeddings and narrows a catalog of more than 170 tools down to roughly 15 relevant to the current request, then hands those to a context-isolated sub-agent. The model never sees the full catalog. This is retrieval applied to capability rather than knowledge, and it comes with a trade-off the post does not dwell on: a retrieval miss at the tool-selection stage is invisible to the sub-agent, which simply cannot do the thing the user asked. Teams adopting this pattern inherit a new evaluation problem — measuring tool recall — on top of task success.

Decoupling the agent from the UI

A second structural issue Postman describes is that many client APIs were implicitly coupled to interface state. Tools that modified a request needed that request to be open; other tools opened tabs as a side effect. The agent ended up imitating a human clicking through the product instead of operating on data. Postman says it is actively decoupling tools from tabs, with its Native Git feature making heavy use of that approach, and that Agent Mode can now send requests in the background without an open tab — though user approval is still required.

This is the least glamorous and probably most transferable lesson. Any product with a decade of accumulated front-end assumptions will have the same coupling, and no amount of prompt engineering fixes it. The work is API refactoring, and the post is candid that it is ongoing rather than finished.

Schema-aware reads instead of a tool per question

For structured data such as the API Catalog — service uptime, test results, endpoint response times — Postman collapsed multiple narrow views into a single query tool. Given the schemas of the underlying ClickHouse tables, the agent generates SQL with joins and filters directly. The post includes an example query aggregating error rates and p95 latency over a seven-day window.

The argument is that engineering effort shifts from building one tool per question to modeling the data well once, after which the agent can answer questions nobody enumerated in advance. That scaling curve is genuinely better. The post does not address the usual costs of giving a model query access: runaway queries, row-level permissions, and the risk of a syntactically valid query that answers the wrong question. Postman notes production testing and monitoring remain necessary, which is a general caveat rather than a specific control for this pattern.

Context, not capability, was the bottleneck

Postman says missing or incomplete context caused more failures than missing tools. Serializing the existing interface data model did not help, because those objects were shaped for rendering and transfer. The team built dedicated context handlers that distill each entity into what an agent needs, split across two kinds of input: broad, shallow background context gathered automatically and minified, and deep, focused context the user explicitly selects and routes through a per-entity handler.

The next problem was truncation. Fields like request descriptions, OpenAPI specifications and payloads are open-ended user data that crowds the window. Postman says it is exploring a filesystem-backed approach so individual handlers do not each need custom truncation and expansion logic. That exploration is unfinished, which is a useful signal: context budgeting at this scale appears to still be an unsolved engineering problem even for a team shipping to a very large developer base.

Product knowledge is handled separately through a RAG knowledge base, seeded from Postman's Learning Center into concise feature-specific articles and selected at runtime based on query and context — injecting the mock server article when a user selects a mock server, for example. Postman frames this as a documentation discipline: ship Agent Mode docs with new features, or the knowledge base rots.

The Bedrock layer: routing and caching

Agent Mode runs on Amazon Bedrock, which Postman uses for managed access to Anthropic Claude models. The post frames inference as a routing-and-caching problem rather than a single model choice. Faster models serve high-volume, latency-sensitive interactions; larger models handle complex reasoning. Moving between supported Claude models is described as mostly a configuration change.

For throughput, Agent Mode uses cross-Region inference profiles passed as the modelId in Converse or InvokeModel calls. Geographic profiles route only among supported Regions in a defined geography such as the US or EU; global profiles route worldwide for maximum burst capacity. The post is careful to note this does not mean inference runs inside Postman's own AWS environment — Bedrock processes requests in eligible Regions, encrypted in transit and at rest. AWS states Bedrock does not use prompts and completions to train AWS models or share them with third parties, and Postman says it has configured zero data retention for supported Agent Mode models, with availability model-dependent.

Caching is tiered. The near-immutable core — system prompt, agent instructions, core tool definitions — uses a one-hour checkpoint; more variable context uses a five-minute checkpoint that refreshes on hit. Bedrock requires the longer TTL to appear first. Postman points teams at the cacheReadInputTokens and cacheWriteInputTokens usage fields to verify behavior. No cost savings or latency figures were disclosed, which is the most notable omission in a section about managing cost and latency.

Oversight and what's unproven

Agent Mode requires user approval before any action that modifies application state, and Postman uses Amazon Bedrock Guardrails to redact personally identifiable information before prompts reach the model, an option enterprise admins enable in guardrail settings. Those are real controls, though approval gates also cap how much autonomy the agent can actually exercise — the demo scenario described, opening a pull request and proposing next steps, still routes through a human.

The post gives no adoption, accuracy, latency or cost metrics for Agent Mode in production, and Postman's implementation is proprietary with no public sample repository. The 40 million developer figure describes Postman's community, not Agent Mode usage. Read it as an architecture narrative with one useful empirical data point, not as evidence of outcomes.

What to watch

• Whether the ~40-tool degradation threshold holds up in other teams' testing, or turns out to be sensitive to tool naming, description quality and model generation.
• How Postman measures tool-retrieval recall — narrowing 170+ tools to ~15 introduces a silent failure mode that task-level success rates will not surface.
• Whether the filesystem-backed context approach ships, and whether it generalizes beyond Postman's entity model.
• Whether AWS or Postman publishes concrete prompt-caching economics; tiered TTLs are only worth the complexity if the cache-write premium is amortized, and no numbers were given.
• How enterprise buyers treat the distinction between geographic inference profiles and true data residency, given that processing happens in AWS Regions rather than the customer's own environment.