// LMCA dataset page — single-column editorial layout. function Todo({ children }) { return {children || "TODO"}; } // LMCA score vs. Text Arena scatter (reuses window.ScatterChart from dtbench-chart.jsx). // Draws directly from the main leaderboard data (data/cri-scores.csv via data.js): // a model is plotted when it has both an `elo` and an LMCA score. function LmcaEloChart() { const LMCA = "Argument evaluation (LMCA)"; const data = (window.MODELS || []) .filter((m) => m.elo != null && m.scores[LMCA] != null) .map((m) => ({ name: m.name, x: m.elo, y: m.scores[LMCA], org: m.org })); return ( v} yFmt={(v) => v} xTipFmt={(v) => Math.round(v)} yTipFmt={(v) => v.toFixed(1)} showR refLine={[{ y: 85, label: "Estimated ceiling (85)" }]} caption={<>More capable models are better at judging argument quality. The y-axis is the LMCA score — the correlation between the human “overall” ratings and each model's “overall” ratings, rescaled so that the score is the correlation times 100 (our estimate of ceiling performance is a correlation of around 0.85, i.e. a score of 85). The x-axis is the model's Text Arena Score. Text Arena ranks AI models based on blind human preference votes. The shaded band is the trend line's 95% Confidence Interval.} /> ); } window.LmcaEloChart = LmcaEloChart; function LmcaPage() { const contributors = ["Emery Cooper*", "Caspar Oesterheld*", "Chi Nguyen", "Alexander Kastner", "Alex Mallen", "Ethan Perez"]; const renderArxiv = (variant) => (
arXiv (update planned)
); return (
LMCA
{renderArxiv("top")}

Language Model Conceptual Argumentation

{renderArxiv("inline")}

“When people ask me what a good dataset for hard-to-verify reasoning looks like, I show them LMCA.”

Joe Benton, then Anthropic Scalable Oversight team lead

LMCA (Language Model Conceptual Argumentation) is our gold-standard conceptual reasoning benchmark. It's a dataset of carefully curated, fully expert-rated conceptual arguments on a diverse range of topics including decision theory, philosophy, and AI safety. It consists of {(window.LMCA_STATS.positionTexts || 0).toLocaleString()} position texts with {(window.LMCA_STATS.ratedCritiques || 0).toLocaleString()} arguments against these position texts. Each argument is rated by at least one expert, for a total of {(window.LMCA_STATS.expertRatings || 0).toLocaleString()} expert ratings.

We use LMCA to measure how good models are at evaluating arguments against position texts by comparing their ratings to the expert ratings. An upcoming extension will use the dataset to measure how good models are at writing arguments against position texts.

Why argumentation?

Progress in conceptual domains such as philosophy relies on informal argumentation. So a model's ability to make and evaluate conceptual arguments is a key measure of its ability to make progress on conceptual questions.

Further, while the bottom-line answer to some conceptual questions is contested, people often agree on whether an argument is a good or bad contribution to the debate. Focusing on argument quality allows us to measure and improve models' conceptual reasoning despite lack of ground truth.

Example

Position text

AIs we train can turn out to be schemers or non-schemers. Schemers can have “a wide variety of goals,” while the motivations of a non-schemer must be benign by definition. Since there are “more” possible schemers than non-schemers, we should expect training to produce schemers most of the time. In Carlsmith's words:

  1. The non-schemer model classes, here, require fairly specific goals in order to get high reward.
  2. By contrast, the schemer model class is compatible with a very wide range of (beyond episode) goals, while still getting high reward…
  3. In this sense, there are “more” schemers that get high reward than there are non-schemers that do so.
  4. So, other things equal, we should expect SGD to select a schemer.

This argument is a counting argument.

Argument against the position text

Counting arguments often yield absurd conclusions. For example, we can construct this analogous argument:

  1. Neural networks must implement fairly specific functions in order to generalize beyond their training data.
  2. By contrast, networks that overfit to the training set are free to do almost anything on unseen data points.
  3. In this sense, there are “more” models that overfit than models that generalize.
  4. So, other things equal, we should expect SGD to select a model that overfits.
{(() => { const dims = ["Centrality", "Strength", "Correctness", "Clarity", "Dead weight", "Single issue", "Overall"]; const emery = { rater: "Emery Cooper", stage: "", vals: [1.0, 0.8, 1.0, 1.0, 0.0, 1.0, 0.8], comments: [ ["Strength", "It's not clear that this is exactly analogous, but it seems intuitively somewhat analogous and it is clear from this that the original argument has to much more carefully justify itself."], ], }; const chiPre = { rater: "Chi Nguyen", stage: "pre-discussion", vals: [1.0, 0.57, 0.9, 1.0, 0.0, 1.0, 0.57], comments: [ ["Strength", "Gives a good counterargument and grounds for skepticism but a stronger critique would engage a bit with the question of why the counting argument for SGD doesn't work and that there aren't very SDG-specific reasons that are unlikely to apply to other cases, so we know that it's a genuine analogy. (And even if it is, it's still nice to know *why* the argument fails) Also, I assume the reason why counting arguments don't work on SGD is because not all else equal. So I guess it depends on whether one thinks the original argument should be seen as an argument for scheming or just as a very weak strict all else equal argument."], ["Correctness", "A little minus because I'm not sure the conclusion is true and because I assume the empirical facts claimed are true but don't actually know"], ], }; const chiPost = { rater: "Chi Nguyen", stage: "post-discussion", vals: [1.0, 0.9, 0.9, 1.0, 0.0, 1.0, 0.7], comments: [ ["Strength", "I was previously subtracting points for not being elucidating *why* counting arguments don't work rather than just assessing how much evidence the critique is against counting arguments and so being highly unsatisfying. But after discussion with Em, I see that that I shouldn't subtract strength for this."], ["Overall", "Subtracting from strength*centrality because it's an unsatisfying argument that doesn't help the reader advance their understanding super much."], ], }; // Merge Chi's two ratings into one row: changed dimensions show old → new. const chiMerged = { rater: "Chi Nguyen", stage: "revised after discussion", vals: chiPre.vals.map((v, i) => (v === chiPost.vals[i] ? chiPost.vals[i] : { from: v, to: chiPost.vals[i] })), }; const tableRows = [emery, chiMerged]; const commentGroups = [emery, chiPre, chiPost]; const fmt = (n) => n.toFixed(2); const renderCell = (v) => { if (typeof v === "number") return fmt(v); return ( {fmt(v.from)} → {fmt(v.to)} ); }; return ( <>
Expert ratings
{["Rater", ...dims].map((h, i) => ( ))} {tableRows.map((r, ri) => ( {r.vals.map((v, i) => ( ))} ))}
{h}
{r.rater}
{r.stage &&
{r.stage}
} 		{renderCell(v)}
Rater comments
{commentGroups.map((r, ri) => (
{r.rater}{r.stage ? " · " + r.stage : ""}
{r.comments.map(([dim, txt], ci) => (

{dim}. {txt}

))}
))}

Each dimension is scored 0–1; struck-through values are Chi Nguyen's pre-discussion scores, revised upward after discussion between raters.

); })()}

Rating methodology and validation

We carefully rated arguments by hand according to a detailed rubric. Arguments are rated on seven dimensions, scored between 0 and 1: centrality, strength, correctness, clarity, dead weight, single issue (whether the argument focuses on a single issue with the position text), and overall. All but 16 arguments are rated by at least Emery Cooper. To ensure quality, we discuss large disagreements between humans and add revised ratings. We also check large disagreements against a panel of frontier models for human error.

We also validated our ratings on a set of around 50 arguments that were each rated by between 4 and 6 people. After rating independently, all raters discussed ratings extensively, spending 7–8 hours total in discussions. Based on our results on this set, we believe models are still somewhat far from ceiling performance. We discuss our results in more detail below.

We currently conservatively estimate that ceiling performance is a correlation with human ratings of around 0.85 (see box below). In the near future, we intend to get more ratings from additional raters, and further refine our estimates of ceiling performance.

Rating validation

By looking at the set of arguments that are rated by both our main rater (Emery Cooper) and one of our secondary raters (Caspar Oesterheld), we obtain rough estimates of ceiling performance. Comparing Emery Cooper's latest ratings (which may be after discussion in some cases) to Caspar Oesterheld's first ratings (which are always before any discussion) gives a Pearson's correlation of 0.87 ± 0.06, compared to a correlation of 0.67 ± 0.09 between Fable 5's and Emery's ratings.

We can also loosely estimate ceiling performance using the set of arguments that 4–6 people rated, with the caveat that this set is quite small.

One way to do this is to compare Emery's pre-discussion ratings with the post-discussion average excluding Emery's rating — the latter being a proxy for ground truth. (Note that most of the dataset has been rated only by Emery, and this provides an estimate of the best possible correlation against Emery's ratings.) Based on this, we estimate ceiling performance to be 0.94 ± 0.06. By contrast, Fable 5 has a correlation of 0.71 ± 0.14 with Emery's pre-discussion ratings on this set. Meanwhile, Fable 5 had a correlation of 0.73 ± 0.13 with the final average of everyone's ratings.

The above metric might overestimate ceiling performance if people converge spuriously toward Emery's ratings during discussion. We can form a more conservative estimate by comparing Emery's pre-discussion ratings with the average pre-discussion rating of others, since these are fully independent. This is conservative, since the average of pre-discussion ratings is even further from a perfect estimate of “ground truth” ratings. Doing this, we obtain the following more conservative estimate of ceiling performance: Emery's pre-discussion ratings had a correlation of 0.86 ± 0.07 with the pre-discussion average of other raters.

); } window.LmcaPage = LmcaPage;