Professional analysts working with an AI agent went along with the agent's randomly assigned lean on 86% of analytical choices, according to a new study from Anthropic. That is far above the roughly 50% expected by chance. The Alignment team study, published October 27 by Amaç Herdağdelen, Hailey Joren and Alex Tamkin, tested whether experts who hand work to an agent still make the decisions that matter.
Anthropic recruited 61 professional analysts from a pre-screened pool, all with causal-inference knowledge, Python or R skills and hands-on observational analysis experience. Eleven chose not to use the agent, leaving 50 in the main analysis. Each assessed whether a fictional farming cooperative's voluntary advisory program raised crop productivity enough, roughly 3% to 4%, to justify expanding it.
The analysts made four open choices: the outcome measure, the comparison method, which farms to cover and how to weight them. They then recommended rolling the program out, discontinuing it or extending it for a year. The agent was Claude Opus 4.8 running in Claude Code at High effort. A coin flip secretly set which option it favored on each decision, and it was told to make the best honest case for that lean. The median task took 65 minutes.
Key facts
- 01Analysts matched their Claude agent's randomly assigned lean on 86% of analytical choices, versus roughly 50% expected by chance.
- 02The agent's lean moved final recommendations to roll out in 54% of cases, versus 36% when it leaned the other way.
- 0377% of analysts repeated the agent's arguments, but only 15% credited the agent.
- 04Without AI, 6 of 11 analysts made serious errors, versus 3 of 50 with the Claude Opus 4.8 agent.
- 05Anthropic's Alignment team published the 61-analyst study on October 27, 2026.
Agreement held across all four decisions, at 92%, 82%, 81% and 90%. Where analysts had already stated a preference, they switched 60% of the time when the agent disagreed and 6% of the time when it agreed. Anthropic notes that figure rests on 38 cases, with a 95% confidence interval of 24 to 82 points.
The leans carried through to the bottom line. When the agent leaned toward expansion, analysts recommended a rollout in 54% of cases, compared with 36% when it leaned the other way. A model using only the four leans predicted 82% of final recommendations, versus 61% for simply guessing the most common answer.
Most analysts did not notice the influence. 77% repeated their agent's arguments, but only 15% credited the agent. 52% rated the final recommendation as more their own than the agent's, yet they were influenced about as much as those who gave the agent more credit. Experience, expertise and education did not predict deferral, and an apparent age effect disappeared after adjusting for multiple comparisons.
The agent also made the work better and faster. Without AI, 6 of 11 analysts, or 55%, made serious errors that invalidated their results, compared with 3 of 50, or 6%, of those using the agent. Non-users took a median 90 minutes versus 65. Anthropic cautions that the two groups differed in experience, so the gap may not be caused by the agent alone.
The concern is oversight. Judgment calls of this kind can shift estimates by several percentage points, which matters when a decision threshold is narrow, and errors in them are hard to catch with standard checks. "Keeping a human in the loop assumes that the human's judgment provides an independent check," the researchers wrote, adding that human review of AI-driven research, including AI safety work, is weakened if the reviewer's judgment is effectively the agent's.
Anthropic proposes three fixes. Teams should build ways to measure deferral in everyday work and better understand models' default preferences. Agents could present the strongest case for each option or ask experts for their view first, raising only the most important decisions to avoid overload. Organizations should decide which decisions require a person's own informed judgment, give people time to form it and protect those skills.
The researchers flag limits. The agent's leans were deliberately strengthened, so real-world effects may be smaller, and the study covered one task, one field, one model and a one-hour window. Results are averages, and the study did not measure how strongly analysts held their own preferences.
Working on something we should cover, or seeing a story we missed? Send leads, documents, or feedback to hello@aichatdaily.com. For sensitive tips, see our secure tips page for Signal and PGP options.
Spotted an error? Email hello@aichatdaily.com with the URL and the issue, or read our full corrections policy.
