Skip to main content
Live
Main content

OpenAI puts Codex agents to work fixing ChatGPT slowdowns

Codex agents ran more than 1,700 investigations in three months and found root causes in 600+ cases, with engineers still approving every fix.

Jaeden Schafer
Editor in Chief · · 5 min read
OpenAI logo

OpenAI has detailed Performance Factory, an internal system that hands ChatGPT performance regressions to Codex agents for investigation. In just over three months, the system's Regression Workflow ran more than 1,700 investigations and identified root causes in more than 600 cases, according to OpenAI.

That works out to roughly 17 investigations a day. About 400 of them reached a proposed fix, and around 15 fixes shipped as exact Codex-written pull requests. Agent suggestions also influenced or matched dozens of other fixes that engineers shipped. Engineers approve every merge, and the system is an internal tool rather than a product launch.

Before the workflow existed, OpenAI's performance engineers had built more than 30 performance tools and leaned on CPU and allocation profilers. Investigations often meant copying an alert into Codex and pasting the findings into Slack. The team then captured its investigation procedures in a reusable Codex skill called "perf-forensics."

Key facts

  • 01OpenAI's Performance Factory ran more than 1,700 Codex investigations of ChatGPT performance regressions in just over three months.
  • 02The agents identified root causes in more than 600 cases, an average of about 17 investigations a day.
  • 03About 400 investigations produced a proposed fix, and around 15 shipped as exact Codex-written pull requests.
  • 04An adversarial Codex reviewer sends back about 25% of diagnoses; a manual audit graded 70% of 100 root causes accurate.
  • 05Engineers approve every merge, and a separate agent verifies recovery to baseline after each deployment.

Performance Factory is built from four parts. A Task Tracker holds task states, transitions and human-input steps, which engineers define in configuration files and Markdown prompts. A Dispatch Loop checks for ready work and starts a Codex session. Cloud Execution runs those sessions inside OpenAI's network with access to code, metrics, logs and diagnostic tools, while Subscriptions wake paused tasks when an event arrives, such as a GitHub review comment or a completed deployment.

Tasks are persistent, so findings live outside any single agent's context. A service identity limits what each agent can do and tracks its actions. When a task needs code review or a deployment, it pauses and resumes once the matching event fires.

An investigation starts with a performance alert. Codex establishes which metrics worsened, when the change began and which requests were affected, and it separates requests that actually got slower from a shift toward serving more requests that were already slow. Its summary for engineers includes the impact, the suspected cause, a proposed repair, charts and traces. Engineers can edit the proposed code or write their own fix.

A second, adversarial Codex reviewer checks each diagnosis and sends about a quarter of them back. In one case, an investigation blamed failing ChatGPT conversations on image-download errors; the reviewer noticed that a file-location service had recovered while image requests kept failing, and reopened the case. A manual audit of 100 root-cause suggestions graded 70% of them accurate.

After a fix is deployed, another agent checks whether performance recovered to baseline. If the problem persists, the task goes back for further investigation. One shipped fix removed repeated processing of the same conversation.

Related · from this week
XPRIZE Wildfire hands out $11M, but no team wins either grand prize
Jaeden Schafer · 5 min read →

OpenAI's performance team said Codex does best when a regression's cause is clear and the fix is relatively contained. Architectural problems, such as a cache that speeds up responses but leaves stale data, remain harder, and agents sometimes stopped at the first plausible explanation even when the evidence did not fully support it.

OpenAI said it is now exploring workflows that compare performance across releases and flag inefficient autoscaling.

ShareXLinkedInEmail
AI Box

Every AI model. One chat.

The latest models from ChatGPT, Claude, Gemini, Sora, ElevenLabs — 80+ models in a single chat. Compare answers side by side. Pick the best one every time.

  • ChatGPT, Claude, Gemini, Grok, DeepSeek — in one chat
  • Generate images & video with Sora, Veo, Ideogram
  • Compare any two models side by side
  • From $8.99/mo · 80+ models, all included
Try AI Boxaibox.ai
Trusted by 3,000+ teams
Got a tip?

Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.

Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.

AI Box Daily briefingFree · Daily · No fluff

Stay ahead of everyone in AI.

The tightly edited AI news email engineers, founders, and investors actually open. One email. Every weekday. Five minutes to finish.

Loved by 10,000+ AI professionals
Free forever. Unsubscribe with one click.

The briefing read inside teams at

Keep reading

More from News

XPRIZE Wildfire hands out $11M, but no team wins either grand prize
News

XPRIZE Wildfire hands out $11M, but no team wins either grand prize

Finalists detected fires from space and struck them with drones inside 10 minutes — none put the fire out.

Jaeden Schafer5 min read
Nvidia logo
News

Inside NVIDIA's validation lab, where Rubin GPUs get broken before customers can

Validation engineer Sakeena Fiza describes the hunt for hardware failures on boards with tens of thousands of components and racks approaching 500,000.

Jaeden Schafer4 min read
OpenAI logo
News

OpenAI enlists nine elite mathematicians to clean up its research rollouts

The IAS-hosted AGMAI panel will advise on releasing 100+ open problems OpenAI says its internal model has resolved.

Jaeden Schafer5 min read