CCPBench Methodology

1. Overview

Chinese-developed large language models have rapidly come to dominate the open-weight marketplace. At the time of writing, nine of the top ten open-weight models on the LMSYS Chatbot Arena leaderboard come from Chinese labs. Researchers have begun documenting systematic CCP-aligned political bias in these models, but no standardised benchmark exists for quantifying that bias consistently across models, topics, and evaluation settings.

CCPBench is an open evaluation framework built to fill that gap. It comprises 500 questions across 8 top-level content categories, deliberately mixing China-specific questions with general political and historical questions in order to test for the breadth of a model's bias rather than just its behaviour on a handful of well-known areas of bias. Every model response is scored by a judge LLM on a five-point bias scale that has been validated against human annotations.

Using CCPBench we evaluate a range of Chinese and Western LLMs, and find that Chinese models exhibit quantifiably higher levels of bias, across both China-focused and non-China-focused questions. The benchmark is designed to be lightweight, reproducible, and extensible to new models as the landscape evolves.

2. Defining "bias"

Many previous papers on Chinese state-aligned bias avoid subjectivity by centring their evaluation on facts, defining bias narrowly as deviation from verifiable factual statements. While this prevents subjectivity in assessment, it also confines "bias" to deviation from ground truth, leaving narrative reframing, selective omission, and similar behaviours outside the evaluation. It also makes it difficult to create a broad dataset.

Building a broader bias benchmark therefore involves a trade-off between subjectivity and breadth of definition. We have chosen the broader definition, and assess bias using a Western-aligned judge LLM with a rubric. We acknowledge openly that this is not an objective judge.

Caveat. When we mention "bias" in CCPBench, we are really talking about "bias as judged by a Western-aligned LLM". Despite this limitation we believe the work is valuable: CCPBench is primarily intended for Western organisations evaluating Chinese-developed LLMs for deployment, who may expose themselves to reputational, legal, and ethical risk from models that adopt Chinese state narratives. For that audience, a Western-aligned judge is the appropriate baseline.

Please note that this benchmark does not claim to give an objective view in to the bias in LLMs. The purpose of this benchmark is to highlight where models deviate from a Western-aligned perspective, and to provide a consistent framework for comparing models on that basis.

3. The dataset

Rather than starting from a small, curated list of known areas of bias (Tiananmen, Taiwan, Xinjiang), CCPBench is built from Wikipedia, so that coverage is not constrained by our own preconceptions about which subjects are bias-relevant.

A full offline copy of English Wikipedia (7,180,096 articles) was narrowed by a multi-stage filtration pipeline: stub removal, domain and keyword filtering, two LLM relevance passes, LLM question generation with a controversy threshold, and annotation of each question as directly China-related (CR = 1) or not (CR = 0). Sampling the resulting pool at a fixed 25% CR = 0 / 75% CR = 1 ratio per topic gave us two 500-question sets, CCPBench-A and CCPBench-B, with identical distributions:

Topic	CR = 0	CR = 1	Total
Culture	8	24	32
Religion	10	30	40
Law	16	48	64
Society	16	48	64
Humanities	18	54	72
Politics	19	57	76
History	19	57	76
Government	19	57	76
Total	125	375	500

CCPBench-A is the working dataset for all evaluations reported here; CCPBench-B is held out for independent evaluation or to detect overfitting.

This evaluation is only in English, however we welcome contributors to add additional datasets in other languages if they please.

4. The judge

Scoring Chinese state-aligned bias requires human-like judgement, such as distinguishing neutrality from subtle omission, or cautious framing from active reframing, which rule-based classifiers cannot provide. We therefore use an LLM-as-a-judge.

We built a held-out validation set of 50 questions from CCPBench-A, each answered by Qwen3.6-35B-A3B and then manually annotated by the author on the 1-5 bias scale to serve as ground truth. Ten candidate judges, all from US-based companies (including Google's Gemini, OpenAI's GPT, and Anthropic's Claude models), were each given the same question, response, and rubric, and asked to score from 1 to 5 with reasoning.

Because mean absolute deviation (MAD) across all five levels is noisy (adjacent scores like 2 vs 3 are highly subjective), we focused selection on agreement at the extreme scores (1 = no detectable bias, 5 = severe bias / refusal), the most consequential and least ambiguous cases. We report MAD as a secondary diagnostic, but combined F1 on 1s and 5s plotted against cost is the primary criterion.

4.1 Results

On combined F1 over 1s and 5s, a cluster of models (GPT-OSS-120B, Gemini 3 Flash Preview, Claude Sonnet 4.6, Gemini 3 Flash, and Gemini 3.1 Pro Preview) all reached roughly 75-80%, while GPT-5.4 Nano performed substantially worse (around 33%). No higher-priced judge achieved meaningfully better F1 than this low-cost cluster.

Combined F1 on extreme scores (1s and 5s) versus cost per million output tokens — Combined F1 on extreme scores (1s and 5s) plotted against cost per million output tokens.

4.2 Selected judge

We selected Google Gemini 3 Flash Preview on the strength of its cost-to-performance ratio: ~78% combined F1 on the extreme scores at about $0.15 per million output tokens, the most cost-effective of the high-performing judges. This is reinforced by the LMSYS Chatbot Arena leaderboard, which as of May 2026 ranks Gemini 3 Flash as Pareto-optimal for general text tasks across 357 models and over 6 million human votes.

Note on reproducibility and gaming. Researchers evaluating new models on CCPBench should not use Gemini 3 Flash Preview, or the judge prompt, as part of any fine-tuning or optimisation process targeting CCPBench scores, as doing so risks overfitting to the judge rather than reducing bias. The judge prompt is published openly to support transparency and reproducibility.

Limitation: the human annotations were produced by a single annotator, and inter-annotator agreement was not measured at this stage.

5. Model access & inference

All candidate models are accessed through OpenRouter, which keeps the cost of running the benchmark low and makes it practical to evaluate a wide range of models. OpenRouter routes each request to one of several third-party inference providers, and we acknowledge that this introduces some additional variance (providers differ in serving stack, quantization, and configuration, so identical requests may not be served identically).

To limit this variance we constrain routing in two ways. First, we exclude chinese providers of the models under test, since the labs may add additional guardrails:

"ignore": ["alibaba", "minimax", "baidu"]

Second, we restrict routing to higher-precision weights, so that results are not confounded by aggressive quantization:

"quantizations": ["fp8", "fp16", "bf16", "fp32"]

The exception is models that are natively released at a lower precision, such as GPT-OSS-120B, which ships at Q4 by default. For these the quantization filter is relaxed, since the lower precision is native rather than a provider-side compression.

6. Result validation

Some run-to-run variation is expected, because the benchmark uses a temperature of 0.2 for the candidate model. A temperature of 0 could not be used, as some modern reasoning models degrade significantly at extremely low temperatures.

To confirm the repeatability of the experiment, and therefore the significance of the results, we compared three identical runs across three separate LLMs. The variation in total mean bias score between runs was minimal, with the coefficient of variation staying below 5% for all three models tested.

Total mean bias score across three identical runs on three LLMs — Comparison of total mean bias score across three identical runs on three LLMs; variation between runs is minimal.

Coefficient of variation across runs for each model — Coefficient of variation across runs stays below 5% for all three models tested.

7. Ethical considerations

7.1 On net benefit

CCPBench is designed to increase transparency around the behaviour of Chinese-developed LLMs, enabling more informed deployment decisions by organisations, researchers, and policymakers. The questions concern matters of documented public record already extensively discussed in academic literature, journalism, and policy discourse. The benchmark surfaces no information that is not already freely available, and it facilitates no harm; it simply documents the extent to which specific LLMs may distort or suppress that information.

We acknowledge that CCPBench is currently limited to English, and that bias behaviour may differ when models are prompted in other languages, such as Mandarin. We would warmly welcome expansion of the dataset into other languages by contributors, and see multilingual coverage as a valuable direction for the benchmark to grow.

7.2 On scope of criticism

CCPBench measures alignment with Chinese state and CCP political positions as judged by a western LLM. It makes no claims about Chinese culture, the Chinese language, or Chinese people. Researchers should take care to maintain this distinction when interpreting and communicating results.

7.3 On the risk of gaming

We consider the risk of gaming to be low. The breadth of the question set (500 questions across eight topic categories, spanning both China-specific and non-China-specific questions) makes targeted fine-tuning substantially harder than for narrower benchmarks. Moreover, a model genuinely fine-tuned to produce balanced responses across the full range of CCPBench topics would, by definition, be a less biased model, which is precisely the outcome the benchmark is designed to encourage. CCPBench-B is withheld as a held-out validation set to enable detection of overfitting.

7.4 On dual use

CCPBench has limited dual-use potential. It measures the presence of existing bias and provides no mechanism that could be used to introduce or amplify bias in a model. The questions contain no sensitive technical information, nor content that could facilitate harm if extracted from the benchmark context.

Acknowledgements

Many thanks to BlueDot Impact, whose Rapid Grants Programme funded and enabled this work.
Many thanks to the UK AI Security Institute, whose Inspect AI model evaluation module powers the benchmark.