Sources & methodology

Public AIpediya Rating always uses BALANCED. Contexts describe comparable tasks; there is no universal ranking across model types. Quality-first and economy profiles remain internal engine options for future collections.

The canonical master stores exact model, configuration, task, runner, numeric result, date and source. Independent external results and Arena preferences are separate from weaker developer reports. Nonpublic research, composites, superseded duplicates and revoked permissions are excluded. AIpediya aggregates external results and does not operate an evaluation lab.

The original frozen capability basket, item coefficients and scale are unchanged. New approved items require at least five exact frozen-theta anchors and append-only ridge binding. An unbound item may be stored in master but never silently scored. Arena Bradley-Terry ratings and Agent IPS are separate metrics with separate frozen transforms.

Family cap is 1.5 effective observations; Epoch cap 6, other runner cap 2. Arena keeps its cap of one effective observation and 25% of precision. Developer evidence is weaker and has a 20% precision share cap. The engine chooses one exact configuration by objective family coverage, then effort. Different efforts are never pooled; an unspecified effort is not inferred.

Arena comes only from the official lmarena-ai/leaderboard-dataset on Hugging Face, with CC BY 4.0 attribution. Raw and style-controlled versions and aggregate/subsignals are not double counted. LiveBench uses officially published task results, with the official release/category definitions. It is a separate objective family and independent external runner. Only subtasks are imported; category and global averages are excluded.

Standard hosted API prices use the existing frozen anchors and comparable units. A confirmed permanent free hosted API may have C=1. Local weights without verified comparable TCO have economics N/A/self-hosted. Resource definitions depend on context: LLM token context, embedding input limit, TTS input limit, STT audio duration, image resolution and video duration. Unconfirmed applicable resources remain missing. No new arbitrary resource anchors are created to raise scores.

Each component is applicable-known, applicable-missing or not applicable. N/A has no estimate and zero weight, and is removed from the coverage denominator. Remaining BALANCED weights are normalized from 78/16/6. Missing applicable components keep their original neutral estimate and wide uncertainty. N/A and unknown are not interchangeable.

A qualified Overall model needs two non-Arena families in two domains. A context needs sufficient models, families and external runners. Specialized contexts additionally require validated distinctiveness. Storing new results does not itself make a context Ready. Unapproved SWE-bench/CODE research remains excluded.

Displayed Rating is the central composite 100 × Q^wq × (0.2+0.8C)^wc × (0.3+0.7K)^wk over applicable components. Every snapshot stores its mask, exact components, base/effective weights and explicit adjustment, currently zero. Runtime verifies reproduction. The tenth percentile is a separate conservative confidence/frontier estimate and never replaces the displayed score.

Precision describes statistical uncertainty after misfit and early-evidence adjustments. Support counts independent external runner paths: A at least three, B two, C fewer. Developer reports never become independent paths. Coverage is the fraction of relevant quality, applicable price/resource, exact configuration, external runner diversity and breadth. N/A is excluded; missing evidence affects confidence and the asterisk.

Freshness uses the snapshot data cutoff, not the request clock. Price/API availability thresholds are 30/60 days, resource 180/365, configuration 90/180. Expired required facts become unknown. A benchmark is excluded for invalidation or revoked permission, not simply age. Snapshot age is a separate warning.

Exact public rank is limited to qualified models within one compatible context. Coverage below 20% suppresses exact rank and labels the score as based on limited data. Conservative frontier uses the frozen meaningful gap delta theta 0.28 and original eligibility gates. Catalog numeric sorting does not invent a cross-context #N.

Builds use 6,000 deterministic Monte Carlo draws, or 12,000 in the existing threshold zone. Stable Record IDs define seeds. Scientific computation runs offline after canonical master changes; runtime reads validated JSON only. Historical snapshots and original calibration bytes are preserved.

No asterisk means the relevant inputs and corroboration requirements are met. Partial * means corroboration is incomplete; Estimated * means a required input is unknown or quality remains a prior. Unknown does not mean bad. Single-factor quality and per-token economics cannot represent every real task; preference results do not become objective benchmarks.