OpenAI has detailed Performance Factory, an internal system that hands ChatGPT performance regressions to Codex agents for investigation. In just over three months, the system's Regression Workflow ran more than 1,700 investigations and identified root causes in more than 600 cases, according to OpenAI.
That works out to roughly 17 investigations a day. About 400 of them reached a proposed fix, and around 15 fixes shipped as exact Codex-written pull requests. Agent suggestions also influenced or matched dozens of other fixes that engineers shipped. Engineers approve every merge, and the system is an internal tool rather than a product launch.
Before the workflow existed, OpenAI's performance engineers had built more than 30 performance tools and leaned on CPU and allocation profilers. Investigations often meant copying an alert into Codex and pasting the findings into Slack. The team then captured its investigation procedures in a reusable Codex skill called "perf-forensics."
Key facts
- 01OpenAI's Performance Factory ran more than 1,700 Codex investigations of ChatGPT performance regressions in just over three months.
- 02The agents identified root causes in more than 600 cases, an average of about 17 investigations a day.
- 03About 400 investigations produced a proposed fix, and around 15 shipped as exact Codex-written pull requests.
- 04An adversarial Codex reviewer sends back about 25% of diagnoses; a manual audit graded 70% of 100 root causes accurate.
- 05Engineers approve every merge, and a separate agent verifies recovery to baseline after each deployment.
Performance Factory is built from four parts. A Task Tracker holds task states, transitions and human-input steps, which engineers define in configuration files and Markdown prompts. A Dispatch Loop checks for ready work and starts a Codex session. Cloud Execution runs those sessions inside OpenAI's network with access to code, metrics, logs and diagnostic tools, while Subscriptions wake paused tasks when an event arrives, such as a GitHub review comment or a completed deployment.
Tasks are persistent, so findings live outside any single agent's context. A service identity limits what each agent can do and tracks its actions. When a task needs code review or a deployment, it pauses and resumes once the matching event fires.
An investigation starts with a performance alert. Codex establishes which metrics worsened, when the change began and which requests were affected, and it separates requests that actually got slower from a shift toward serving more requests that were already slow. Its summary for engineers includes the impact, the suspected cause, a proposed repair, charts and traces. Engineers can edit the proposed code or write their own fix.
A second, adversarial Codex reviewer checks each diagnosis and sends about a quarter of them back. In one case, an investigation blamed failing ChatGPT conversations on image-download errors; the reviewer noticed that a file-location service had recovered while image requests kept failing, and reopened the case. A manual audit of 100 root-cause suggestions graded 70% of them accurate.
After a fix is deployed, another agent checks whether performance recovered to baseline. If the problem persists, the task goes back for further investigation. One shipped fix removed repeated processing of the same conversation.
OpenAI's performance team said Codex does best when a regression's cause is clear and the fix is relatively contained. Architectural problems, such as a cache that speeds up responses but leaves stale data, remain harder, and agents sometimes stopped at the first plausible explanation even when the evidence did not fully support it.
OpenAI said it is now exploring workflows that compare performance across releases and flag inefficient autoscaling.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.




