Optimize Once, Execute Fast: Why AI Teams Make You Wait, and How to Fix the Org Chart

Created on November 01, 2026

2026   ·   LLM   multi-agent   latency   workflows   NeurIPS   ·   ai

The forty-second problem

Picture a customer-support desk that fields ten thousand similar questions a day. The company’s AI answers each one correctly. It also takes forty seconds to do it. Those forty seconds are paid ten thousand times over, every day, and nobody designed it that way. The system was tuned to get the answer right. The delay came along for free.

This is now a common situation. The most capable AI systems in deployment are no longer single models. They are teams. One model reads a request and plans an approach, another drafts the answer, a third checks it for errors, a fourth makes sure it follows company policy. Each member is a specialist, and the division of labor raises the quality of the result. But the request passes through a sequence of hands, and each hand takes time.

For a one-off task, a slow answer is a minor annoyance. For the recurring workloads where businesses actually deploy AI, such as support desks, IT troubleshooting, and code review, the same kind of request arrives over and over, and the delay compounds. A workflow that is a few seconds too slow becomes hours of customer waiting by the end of the week.

Our paper, accepted at NeurIPS 2026, asks a simple question about these AI teams: how much of the waiting is necessary, and how much is just a bad org chart?

Why AI teams are slower than they need to be

Each time a workflow calls a model, it waits for that model to think, and on a busy day it also waits in line behind other requests. In a workflow designed as a chain, every step waits for the step before it. The total time is set by the longest sequence of steps that truly depend on one another. Project managers call this the critical path.

The trouble is that many workflows are longer than their critical path needs to be. A step that checks formatting does not need to wait for a step that checks arithmetic, yet the workflow makes it wait. A second review adds little that the first did not, yet it sits in the sequence. Think of a restaurant kitchen where the cook plating one dish waits for a colleague chopping vegetables for a different dish. Nothing about the food requires the wait. The layout of the kitchen does.

A toy five-step workflow before and after reorganization. The two checks never needed each other's output, and the second style check never needed to exist. Six seconds and three model calls disappear; the answer does not change.

Researchers have built tools that design these AI workflows automatically, but nearly all of them optimize for one thing: getting the answer right. Some also try to reduce computing cost. Very few look at the clock.

What we did

We treat a workflow as a map of who waits for whom, and then edit the map. There are two kinds of edit.

The only two moves the optimizer is allowed to make. Both come with a proof that they can shorten the critical path but never lengthen it.

The first edit lets two steps run side by side when one does not actually need the other’s output. The second removes a step that adds time but not value. Each edit comes with a mathematical guarantee: it can only shorten the longest chain of dependent steps, never lengthen it, and no step loses an input it previously had. We also show that these two edits are enough, in principle, to reach any sensible workflow design from a simple starting point.

Speed alone is not the goal. To keep quality from slipping, we train a small judge that predicts how well a candidate workflow will perform on the task, and we block any edit the judge expects to hurt accuracy. Because the search runs on the map itself rather than by calling the models, finding an improved design takes one to two seconds.

The method works in two modes. It can design a workflow from scratch. Or it can take a workflow a company has already built, including one produced by another automated tool, and make it faster without changing how it was built. That second mode is the one most relevant to firms with AI systems already in production.

What the numbers show

We tested the approach on seven standard benchmarks spanning code generation, math word problems, and general-knowledge questions, against nine alternative ways of organizing an AI team. The reorganized workflows were the fastest on six of the seven benchmarks and had the best accuracy ranking overall. Speed did not come at the expense of correctness.

Seconds per query across seven benchmarks. The orange line is the naive chain where every step waits; gray lines are the strongest prior automated designs; blue is ours.

The starkest comparison came when we swapped in one of the most capable current AI models as the team’s engine. On coding tasks, the reorganized team answered in about 14 seconds. Two widely used alternatives took 40 and 44 seconds and were slightly less accurate. On general-knowledge questions, the gap was similar: about 20 seconds against 33 and 53.

With a frontier model powering every agent, the reorganized workflow is both the fastest and the most accurate on all three tasks.

Used as a bolt-on, the method trimmed three to eight seconds per query from workflows designed by four other approaches, with no loss in accuracy on average. Because redundant steps are removed, the reorganized workflows also consumed less model computation per query, so they were cheaper to run as well as faster.

Applied to workflows that other methods designed, the two edits still shave seconds off, and accuracy nudges up rather than down.

Why “optimize once”

The restructuring has a one-time cost: training the judge and running the search. For the slowest starting designs that cost is recovered after a handful of queries, and every query after that is pure savings. For a workload that repeats thousands of times a day, the arithmetic is not close.

Cumulative waiting time with and without the one-time optimization, using the coding benchmark's numbers. The setup pays for itself before the fourth query.

Why it matters beyond the paper

As companies move from AI pilots to AI in daily operations, response time stops being a technical footnote and becomes a product variable. It shapes whether customers stay on the line, whether engineers trust an automated troubleshooter, and what the computing bill looks like at the end of the month. The design of an AI workflow, which steps run, in what order, and which wait for which, is as much a management decision as an engineering one.

Accuracy and speed are usually presented as a trade-off, where faster means sloppier. Our results suggest that much of the delay in deployed AI teams is organizational slack rather than necessary work, and that it can be removed by reorganizing rather than by cutting corners. The same logic that lets an operations manager shorten a production line applies to a line of AI models.

The work has limits we are open about. It assumes a fixed roster of AI specialists and a stable mix of tasks. Workloads that shift over time, or teams whose members change, will need the method to adapt rather than be rebuilt, and that is where we are heading next.

About the paper

“Optimize Once, Execute Fast: Latency-Aware Multi-Agent Workflow Learning for Recurrent Queries,” by Enpei Zhang (Dartmouth), Feiyu Qu (Dartmouth), Zheng Huang (Dartmouth), Dawei Zhou (Virginia Tech), Elynn Chen (NYU Stern), and Yujun Yan (Dartmouth). NeurIPS 2026, the 40th Conference on Neural Information Processing Systems.

This work was supported in part by the National Science Foundation under grant DMS-2412577.