AWS Details the Plumbing Behind Its Internal Executive AI Assistant

AWS has published an engineering breakdown of NarrateAI, an internal conversational analytics assistant that the company says serves more than 4,000 AWS executive leaders. The post, the second in a series, focuses on five quality assurance techniques built on Amazon Bedrock that AWS says together deliver roughly 99 percent numerical accuracy while streaming answers in real time. It is one of the more specific accounts of how a large vendor handles the unglamorous part of shipping an LLM application: keeping it available, keeping it fast, and keeping it from making up numbers in front of an executive audience.

The problem AWS says it was solving

NarrateAI is described as a two-layer system on Amazon Bedrock AgentCore: a batch layer that generates narratives automatically, and a conversational layer for live interaction. The new post covers only the real-time layer.

The framing is worth noting because it explains the design choices. AWS says the assistant is used during live business reviews, where a hallucinated metric or a slow response carries immediate professional consequences. The post cites a concrete failure mode: reporting $441M when the real figure is $414M, a transposition small enough to survive a skim and large enough to distort a resource decision. That risk profile justifies spending latency and compute on validation in a way that a consumer chatbot would not.

Routing by data volume rather than perceived difficulty

The first technique, adaptive pipeline orchestration, routes each query based on the total volume of retrieved document sections rather than any semantic judgment about difficulty. If the aggregated volume falls under an empirically calibrated character threshold, everything is concatenated into a single context and handled with one LLM call. Above the threshold, sections are packed into batches that respect document boundaries, analyzed in parallel, then merged by a consolidation call that resolves conflicts.

AWS reports that roughly 90 percent of production queries take the fast path, finishing in under 25 seconds with time-to-first-token in a few seconds, while the remaining 10 percent take 50 to 75 seconds. With a typical four batches on the slow path, the company puts the blended cost at about 1.4 LLM invocations per query, a 72 percent reduction against always running the multi-pass strategy.

The threshold is set relative to the model's context window, which the post pegs at typically 200K tokens, and calibrated to capture 70 to 90 percent of traffic on the fast path. That calibration is workload-specific, and AWS says as much. Teams copying the pattern are inheriting a tuning problem, not a solution: the economics collapse if your query mix pushes most requests over the line.

Quota arbitrage as an availability strategy

The second technique is the most transferable and the most Bedrock-specific. AWS notes that Bedrock assigns quotas independently per model and per account, so a three-model by three-account configuration yields nine independent quota spaces. NarrateAI attempts the highest-ranked model first and shuffles the list of AWS role ARNs before each attempt using Python's random.shuffle(), spreading load roughly evenly across accounts without a central coordinator.

Rather than backing off on throttling, a detector catches ThrottlingException, ServiceQuotaExceededException and TooManyRequestsException and immediately retries a different model-account pair. Fresh credentials via AWS Security Token Service take 100 to 200 milliseconds, which AWS argues is negligible next to exponential backoff. The whole thing is packaged as a custom Strands model provider that drops in for the standard BedrockModel.

AWS says that over a six-month deployment the approach absorbed traffic surges and reduced user-visible throttling, and that Locust load tests sustained more than 100 concurrent streaming users without failed requests. Two caveats the post does not raise: the throttling improvement is stated qualitatively, with no before-and-after numbers, and 100 concurrent users is a modest bar for a population of 4,000. There is also an operational cost to running multiple accounts, cross-account IAM roles and a model ranking that silently degrades response quality during cascades. Users get the best available model, but they are not told when they did not get the best model.

Evaluating paragraphs while the model is still writing

The third technique addresses a tradeoff most teams hit eventually. AWS says its first implementation validated the complete response before delivering anything, pushing time-to-first-content past a minute. The fix rests on an observation that validating paragraph N does not require paragraph N+1, so evaluation can overlap generation using a producer-consumer pattern: a producer accumulates streamed tokens into paragraphs at double-newline boundaries, a bounded FIFO queue buffers them, and a consumer validates and releases approved text.

Measured across 1,000 production queries and about 10,439 paragraphs using Anthropic Claude Sonnet on Bedrock, AWS reports median time-to-first-content of 13.2 seconds with parallel evaluation, 11.8 seconds with no checks at all, and 100.2 seconds for the sequential baseline. That is an 86.8 percent reduction against sequential, with 1.4 seconds of overhead attributed to evaluating the first paragraph before the pipeline reaches steady state.

The behavior is bimodal. Deterministic checks finish in about 79 milliseconds median, keeping buffer utilization near 0.095. Paragraphs that trigger LLM-based numerical verification take about 2,025 milliseconds, pushing median utilization to roughly 21.5, at which point backpressure throttles the producer until the consumer catches up. AWS presents this as stable rather than broken, and it is, but it means users see stutter precisely on the paragraphs that contain numbers.

Three evaluators, and the gaps around them

The composite framework runs evaluators in parallel against each paragraph behind a shared pass/fail interface, with Amazon Bedrock Guardrails screening inbound queries separately. The three reference evaluators are a regex-based weasel word check that pushes toward quantitative phrasing, a Unicode emoji filter, and a data accuracy evaluator that cascades from exact value matching to string similarity to LLM semantic verification, reserving the expensive stage for cases that need it. Correctors run in a fixed order: text modification, then character stripping, then annotation.

AWS says nearly a quarter of flagged paragraphs tripped two or more evaluators, which it presents as the main argument for the composite design. It does not disclose what share of paragraphs were flagged overall, the false positive rate, or how the roughly 99 percent numerical accuracy figure was measured. Notably, the data accuracy corrector annotates suspected fabrications with "LLM Reasoning" labels rather than suppressing them, shifting the final judgment to the reader.

The extensibility rule is the most reusable idea here: a new evaluator is admissible only if the combined evaluation-to-generation ratio stays below 1. Cheap deterministic checks are free; expensive LLM judges must fire rarely. That is a discipline most evaluation stacks lack.

One structural limit goes unaddressed. Paragraph-level validation cannot catch inconsistencies that span paragraphs, and it cannot verify that a correct number is being used to support a wrong conclusion.

What to watch

• Whether AWS publishes the methodology behind the roughly 99 percent numerical accuracy claim, including how ground truth was established and what the false positive rate on flagged paragraphs looks like.
• Whether any of this ships as Bedrock or AgentCore product surface rather than an internal pattern; the custom Strands model provider in particular looks like a candidate for productization.
• How the cross-account quota strategy holds up as more teams adopt it, and whether AWS treats multi-account quota expansion as a supported pattern or an artifact of how quotas are currently allocated.
• Whether 13.2 seconds to first content proves acceptable in live meetings, or whether the evaluation budget gets cut once the novelty wears off.
• Follow-up posts in the series covering the second verification stage, which the published text does not fully detail, and the batch narrative generation layer.