Artificial Analysis

The site tracks four categories of performance for each model:

Speed — tokens generated per second once the model starts responding. Higher is better. The site measures this against each hosting provider's live API; you can see, for example, that Llama-3.1-70B runs roughly 2× faster on Fireworks than on Replicate, and roughly 4× faster than on a self-hosted vLLM instance.

Latency — time-to-first-token, in milliseconds. This matters most for chat-style applications where users see streaming responses. A model that generates 200 tok/s but takes 2 seconds to start can feel slower than one that generates 80 tok/s and starts in 200ms.

Price — input and output cost per million tokens, separated. The breakout matters because input-heavy and output-heavy workloads land very differently on different price ladders. Claude is roughly even on input/output; GPT-4 charges 4× more for output than input.

Quality — an aggregate of published evals (MMLU, GPQA, MATH, HumanEval, MMLU-Pro, and others) into a single score per model. AA doesn't run its own evals; it collects what the labs and academic groups publish and presents them in one chart.

LMArena measures something specific: which response a human picks in a blind A/B. That's a useful signal — humans are the eventual customer — but it's heavily biased by length, formatting, and tone. A confident, well-formatted, slightly-too-long response often wins on Arena even when the underlying answer is wrong.

Artificial Analysis measures things that don't have those biases: how many milliseconds elapsed, how many cents per million tokens, what score did the model get on a public eval. Less subjective. Less interesting to read. More useful when you're picking a production model.

The two sites aren't substitutes. A team picking a model for a customer-facing chatbot probably wants both — high Arena rank for perceived quality, low AA latency and price for unit economics. A team picking a model for a batch-summarization pipeline mostly wants AA — Arena rank doesn't matter when no one sees the individual outputs.

The single most useful thing on Artificial Analysis is the hosting-provider comparison. Most benchmark sites treat "Llama-3.1-70B" as a single row. AA treats it as a row per provider — Together, Fireworks, Replicate, Anyscale, Groq, Cerebras, sometimes more — with measured numbers for each.

The numbers are surprising more often than not. Groq routinely beats every other provider on speed by 3–5×, but the price per million tokens varies. Together is often the cheapest for a given model but slower than Fireworks. Cerebras is the fastest at the high end but limited to a smaller catalog.

If you're building on top of an open-weights model, the AA hosting matrix is the closest thing to a buyer's guide. The labs' marketing pages rarely break out which provider's numbers they're quoting; AA does.

The quality scores are aggregates, not original evals. If you care exactly how Claude 3.7 performs on, say, GPQA Diamond — go read the Anthropic system card. AA's score is a weighted average of public benchmarks, and the weighting choices are AA's, not yours.

Subjective output quality is harder to capture. AA can tell you that GPT-4o is 30% cheaper than Claude 3.5 Sonnet on output tokens. It cannot tell you which one writes better marketing copy or follows multi-step instructions more reliably. For that you still need to test on your own data.

Speed measurements vary by region and time of day. AA averages across measurements, which is the right approach, but a single point-in-time measurement on a Friday afternoon in EU-West can look very different from a Tuesday morning in US-East. If latency is mission-critical, you should measure on your own infrastructure.

Open artificialanalysis.ai, pick the comparison view, and add the models you're considering. Filter by hosting provider if you have a preferred vendor or are open to switching. The "intelligence index" and "speed" axes are the two views that matter most; "price" matters mostly when you're at scale.

For new model launches, the site usually updates within 48 hours. Treat the first day's numbers as preliminary — providers tune their hosting in the first week and numbers settle by the second.

If you ship AI in production, Artificial Analysis belongs in the same browser tab group as your provider dashboards. It is the only public site that makes hosting-provider differences legible, the only one that breaks input vs. output pricing cleanly, and the only one whose latency numbers are continuously measured rather than vendor-claimed. Free, no login, updated daily — there is no reason not to use it.