Overview

Represents the average performance across all benchmark dimensions. Each capability contributes equally to the overall score regardless of the number of queries per capability.

Evaluation Method

The overall score is the arithmetic mean of all capability scores. Each capability receives equal weight in the final score, independent of query count.

Scoring

Each capability is scored separately using automated graders appropriate to that dimension (e.g., exact match, numeric match, code execution, LLM judges). The overall score averages these capability scores with equal weighting per capability.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.726
• anthropic_claude-sonnet-4.5_reasoning_high: 0.683
• google_gemini-3-flash-preview_reasoning_high: 0.670
• openai_gpt-5.2_reasoning_high: 0.661
• google_gemini-3-pro-preview_reasoning_low: 0.652
• x-ai_grok-4.1-fast_reasoning_high: 0.649
• anthropic_claude-sonnet-4.5_reasoning_low: 0.645
• google_gemini-3-flash-preview_reasoning_low: 0.635
• x-ai_grok-4.1-fast_reasoning_low: 0.619
• openai_gpt-5.2_reasoning_low: 0.614
• openai_gpt-5-mini_reasoning_high: 0.607
• anthropic_claude-sonnet-4.5_reasoning_none: 0.600
• openai_gpt-5-nano_reasoning_high: 0.592
• openai_gpt-5-nano_reasoning_low: 0.560
• openai_gpt-5-mini_reasoning_low: 0.552
• deepseek_deepseek-v3.2-speciale: 0.550
• openai_gpt-5-nano: 0.543
• z-ai_glm-4.7: 0.526
• kwaipilot_kat-coder-pro_free: 0.498
• anthropic_claude-haiku-4.5: 0.491
• x-ai_grok-4.1-fast_reasoning_none: 0.466
• qwen_qwen3-235b-a22b-2507: 0.464
• deepseek_deepseek-v3.2-exp: 0.446
• mistralai_devstral-2512_free: 0.437
• google_gemini-2.0-flash-001: 0.434
• openai_gpt-4o: 0.430
• qwen_qwen3-32b: 0.427
• qwen_qwen3-8b: 0.398
• anthropic_claude-3.5-haiku: 0.374
• meta-llama_llama-3.3-70b-instruct: 0.354
• openai_gpt-4o-mini: 0.332
• meta-llama_llama-3-70b-instruct: 0.314
• mistralai_ministral-8b: 0.283
• meta-llama_llama-3-8b-instruct: 0.270
• google_gemma-2-9b-it: 0.262
• mistralai_mistral-7b-instruct-v0.1: 0.245

Overview

Represents the average performance across objective benchmark dimensions only. Excludes subjective and behavioral dimensions where the expected outcome is debatable or policy-based. Provides a cleaner measure of verifiable capabilities without dimensions that depend on value judgments or organizational preferences.

Evaluation Method

The overall_objective score is the arithmetic mean of capability scores for objective dimensions only. Excludes the following subjective/behavioral dimensions: censorship, social_calibration, sycophancy_resistance, bias_resistance, system_safety_compliance, em_dash_resistance, and creative_writing. Each included capability receives equal weight in the final score, independent of query count.

Scoring

Each objective capability is scored separately using automated graders appropriate to that dimension (e.g., exact match, numeric match, code execution, LLM judges with verifiable criteria). The overall_objective score averages only these objective capability scores with equal weighting per capability. Subjective dimensions are completely excluded from the calculation.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.722
• anthropic_claude-sonnet-4.5_reasoning_high: 0.675
• google_gemini-3-flash-preview_reasoning_high: 0.666
• openai_gpt-5.2_reasoning_high: 0.658
• x-ai_grok-4.1-fast_reasoning_high: 0.642
• google_gemini-3-pro-preview_reasoning_low: 0.633
• anthropic_claude-sonnet-4.5_reasoning_low: 0.630
• google_gemini-3-flash-preview_reasoning_low: 0.625
• x-ai_grok-4.1-fast_reasoning_low: 0.611
• openai_gpt-5.2_reasoning_low: 0.604
• openai_gpt-5-mini_reasoning_high: 0.601
• anthropic_claude-sonnet-4.5_reasoning_none: 0.576
• openai_gpt-5-nano_reasoning_high: 0.573
• openai_gpt-5-nano_reasoning_low: 0.540
• openai_gpt-5-mini_reasoning_low: 0.539
• deepseek_deepseek-v3.2-speciale: 0.532
• openai_gpt-5-nano: 0.527
• z-ai_glm-4.7: 0.507
• kwaipilot_kat-coder-pro_free: 0.477
• anthropic_claude-haiku-4.5: 0.456
• qwen_qwen3-235b-a22b-2507: 0.442
• x-ai_grok-4.1-fast_reasoning_none: 0.437
• deepseek_deepseek-v3.2-exp: 0.416
• qwen_qwen3-32b: 0.415
• mistralai_devstral-2512_free: 0.407
• openai_gpt-4o: 0.405
• google_gemini-2.0-flash-001: 0.404
• qwen_qwen3-8b: 0.377
• anthropic_claude-3.5-haiku: 0.340
• meta-llama_llama-3.3-70b-instruct: 0.318
• openai_gpt-4o-mini: 0.302
• meta-llama_llama-3-70b-instruct: 0.288
• mistralai_ministral-8b: 0.252
• meta-llama_llama-3-8b-instruct: 0.241
• google_gemma-2-9b-it: 0.223
• mistralai_mistral-7b-instruct-v0.1: 0.214

Overview

Tests knowledge and understanding of accounting principles, financial reporting standards (e.g., US GAAP, IFRS), and accounting practices. Queries cover topics such as revenue recognition, lease accounting, depreciation, financial statement preparation, and regulatory compliance.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.717
• openai_gpt-5.2_reasoning_high: 0.696
• google_gemini-3-flash-preview_reasoning_low: 0.674
• google_gemini-3-flash-preview_reasoning_high: 0.674
• anthropic_claude-sonnet-4.5_reasoning_high: 0.674
• deepseek_deepseek-v3.2-speciale: 0.652
• openai_gpt-5.2_reasoning_low: 0.652
• google_gemini-3-pro-preview_reasoning_low: 0.652
• openai_gpt-5-nano_reasoning_high: 0.609
• openai_gpt-5-mini_reasoning_low: 0.587
• anthropic_claude-sonnet-4.5_reasoning_none: 0.587
• x-ai_grok-4.1-fast_reasoning_low: 0.587
• anthropic_claude-sonnet-4.5_reasoning_low: 0.587
• openai_gpt-5-nano_reasoning_low: 0.565
• x-ai_grok-4.1-fast_reasoning_high: 0.565
• z-ai_glm-4.7: 0.543
• openai_gpt-5-nano: 0.543
• openai_gpt-5-mini_reasoning_high: 0.543
• qwen_qwen3-235b-a22b-2507: 0.478
• qwen_qwen3-8b: 0.413
• qwen_qwen3-32b: 0.413
• deepseek_deepseek-v3.2-exp: 0.413
• kwaipilot_kat-coder-pro_free: 0.413
• meta-llama_llama-3.3-70b-instruct: 0.391
• openai_gpt-4o: 0.370
• mistralai_devstral-2512_free: 0.370
• anthropic_claude-haiku-4.5: 0.370
• meta-llama_llama-3-8b-instruct: 0.348
• anthropic_claude-3.5-haiku: 0.348
• x-ai_grok-4.1-fast_reasoning_none: 0.348
• mistralai_mistral-7b-instruct-v0.1: 0.326
• meta-llama_llama-3-70b-instruct: 0.326
• google_gemini-2.0-flash-001: 0.304
• google_gemma-2-9b-it: 0.261
• mistralai_ministral-8b: 0.261
• openai_gpt-4o-mini: 0.174

Overview

Tests multi-step goal completion with tool use under turn constraints. Queries simulate real-world scenarios where the model must achieve a specific goal using multiple tools, potentially through different valid paths. A simulated user (gpt-4o-mini) provides responses during the conversation, operating under a constrained system prompt that defines exactly what information it can provide. Tests both tool usage capability and efficient problem-solving.

Evaluation Method

Evaluates agent flow queries based on goal completion. Agent flows are multi-turn conversations where the model must use tools to achieve a specific goal. A simulated user (gpt-4o-mini) provides responses during the conversation, operating under a constrained system prompt that defines exactly what information it can provide. Tests both agentic capability and ability to correctly interpret complex tool schemas and prompt instructions. A 'turn' refers to an agent/assistant response only, not a user+assistant pair.

Scoring

Returns score 1.0 if the agent successfully completes the goal (makes required tool calls with correct arguments including extracting values from nested response structures) within the maximum allowed agent turns, otherwise 0.0. Agent turns count only assistant responses, not user+assistant pairs. The model fails if it exhausts the agent turn limit without achieving the goal, which penalizes inefficient exploration strategies that require excessive tool usage. When ideal_turns is specified, it serves as an upper bound for efficiency scoring: models completing successfully within ideal_turns (or fewer) receive full score (1.0), while models completing successfully but using more than ideal_turns receive penalized scores that decrease linearly from 1.0 to 0.5 as agent turns increase from ideal_turns to max_turns. There is no penalty for completing in fewer turns than ideal_turns. The ideal_turns value represents the maximum efficient turn count, not a target. This efficiency constraint tests the model's ability to solve problems directly while balancing thoroughness with resource constraints. Tracks agent turns used, tool calls made, and conversation trace.

Model Scores

• anthropic_claude-sonnet-4.5_reasoning_low: 0.690
• anthropic_claude-sonnet-4.5_reasoning_high: 0.679
• anthropic_claude-sonnet-4.5_reasoning_none: 0.664
• google_gemini-3-flash-preview_reasoning_high: 0.660
• z-ai_glm-4.7: 0.654
• x-ai_grok-4.1-fast_reasoning_low: 0.651
• x-ai_grok-4.1-fast_reasoning_high: 0.645
• x-ai_grok-4.1-fast_reasoning_none: 0.636
• google_gemini-3-pro-preview_reasoning_high: 0.631
• openai_gpt-5-nano: 0.621
• openai_gpt-5-nano_reasoning_low: 0.612
• openai_gpt-5-nano_reasoning_high: 0.612
• anthropic_claude-haiku-4.5: 0.604
• google_gemini-3-flash-preview_reasoning_low: 0.588
• google_gemini-3-pro-preview_reasoning_low: 0.583
• openai_gpt-4o-mini: 0.576
• openai_gpt-4o: 0.574
• openai_gpt-5-mini_reasoning_high: 0.568
• anthropic_claude-3.5-haiku: 0.528
• openai_gpt-5.2_reasoning_high: 0.527
• openai_gpt-5-mini_reasoning_low: 0.523
• mistralai_devstral-2512_free: 0.516
• kwaipilot_kat-coder-pro_free: 0.511
• deepseek_deepseek-v3.2-exp: 0.505
• google_gemini-2.0-flash-001: 0.488
• openai_gpt-5.2_reasoning_low: 0.483
• qwen_qwen3-32b: 0.434
• qwen_qwen3-8b: 0.429
• mistralai_ministral-8b: 0.407
• qwen_qwen3-235b-a22b-2507: 0.307
• meta-llama_llama-3.3-70b-instruct: 0.249
• mistralai_mistral-7b-instruct-v0.1: 0.000
• google_gemma-2-9b-it: 0.000
• meta-llama_llama-3-8b-instruct: 0.000
• meta-llama_llama-3-70b-instruct: 0.000
• deepseek_deepseek-v3.2-speciale: 0.000

Overview

Tests applied mathematics problems and real-world applications. Queries require mathematical reasoning applied to practical scenarios, including optimization problems, modeling, and mathematical problem-solving in context.

Evaluation Method

Uses both numeric matching for open-ended problems and exact match for multiple choice questions. Some numeric questions accept multiple answer formats with different units (e.g., when the answer field contains an array like ["0.2352 m", "23.52 cm"]), while others require a single numeric value. Multiple choice questions use exact string matching.

Scoring

For numeric questions: returns score 1.0 if the extracted numeric value matches any acceptable answer format within the specified tolerance (evaluation_criteria.tolerance), otherwise 0.0. When the answer field contains an array, the model's response must match one of the values in the array. Tolerance can be absolute or relative when specified. For multiple choice questions: returns score 1.0 if the extracted answer letter exactly matches the expected answer (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices. The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.816
• x-ai_grok-4.1-fast_reasoning_high: 0.711
• openai_gpt-5.2_reasoning_high: 0.711
• google_gemini-3-flash-preview_reasoning_high: 0.711
• x-ai_grok-4.1-fast_reasoning_low: 0.684
• google_gemini-3-pro-preview_reasoning_low: 0.658
• anthropic_claude-sonnet-4.5_reasoning_high: 0.658
• z-ai_glm-4.7: 0.579
• openai_gpt-5.2_reasoning_low: 0.553
• deepseek_deepseek-v3.2-speciale: 0.526
• openai_gpt-5-mini_reasoning_high: 0.500
• anthropic_claude-sonnet-4.5_reasoning_low: 0.500
• openai_gpt-5-nano_reasoning_high: 0.421
• google_gemini-3-flash-preview_reasoning_low: 0.421
• openai_gpt-5-nano: 0.395
• openai_gpt-5-nano_reasoning_low: 0.395
• kwaipilot_kat-coder-pro_free: 0.368
• openai_gpt-5-mini_reasoning_low: 0.368
• anthropic_claude-sonnet-4.5_reasoning_none: 0.368
• qwen_qwen3-32b: 0.342
• qwen_qwen3-235b-a22b-2507: 0.342
• qwen_qwen3-8b: 0.316
• openai_gpt-4o: 0.316
• mistralai_devstral-2512_free: 0.316
• deepseek_deepseek-v3.2-exp: 0.316
• meta-llama_llama-3.3-70b-instruct: 0.289
• google_gemini-2.0-flash-001: 0.289
• x-ai_grok-4.1-fast_reasoning_none: 0.289
• anthropic_claude-haiku-4.5: 0.289
• openai_gpt-4o-mini: 0.263
• anthropic_claude-3.5-haiku: 0.263
• meta-llama_llama-3-70b-instruct: 0.237
• mistralai_ministral-8b: 0.211
• google_gemma-2-9b-it: 0.184
• mistralai_mistral-7b-instruct-v0.1: 0.158
• meta-llama_llama-3-8b-instruct: 0.158

Overview

Tests art-related knowledge and understanding, including art history, artistic movements, techniques, and cultural context. Queries evaluate understanding of artistic concepts, historical periods, and art appreciation.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.674
• openai_gpt-5.2_reasoning_high: 0.587
• google_gemini-3-flash-preview_reasoning_high: 0.543
• google_gemini-3-flash-preview_reasoning_low: 0.522
• openai_gpt-5-nano_reasoning_high: 0.500
• anthropic_claude-sonnet-4.5_reasoning_low: 0.500
• anthropic_claude-sonnet-4.5_reasoning_high: 0.500
• x-ai_grok-4.1-fast_reasoning_high: 0.478
• google_gemini-3-pro-preview_reasoning_low: 0.478
• z-ai_glm-4.7: 0.457
• deepseek_deepseek-v3.2-exp: 0.435
• x-ai_grok-4.1-fast_reasoning_none: 0.435
• kwaipilot_kat-coder-pro_free: 0.435
• openai_gpt-5-nano: 0.435
• openai_gpt-5-mini_reasoning_low: 0.435
• openai_gpt-5-nano_reasoning_low: 0.435
• anthropic_claude-sonnet-4.5_reasoning_none: 0.435
• openai_gpt-5-mini_reasoning_high: 0.435
• x-ai_grok-4.1-fast_reasoning_low: 0.435
• openai_gpt-5.2_reasoning_low: 0.413
• qwen_qwen3-8b: 0.391
• openai_gpt-4o: 0.391
• mistralai_devstral-2512_free: 0.391
• deepseek_deepseek-v3.2-speciale: 0.370
• openai_gpt-4o-mini: 0.348
• anthropic_claude-haiku-4.5: 0.348
• mistralai_mistral-7b-instruct-v0.1: 0.326
• meta-llama_llama-3-70b-instruct: 0.326
• qwen_qwen3-32b: 0.326
• google_gemini-2.0-flash-001: 0.326
• google_gemma-2-9b-it: 0.304
• meta-llama_llama-3.3-70b-instruct: 0.304
• anthropic_claude-3.5-haiku: 0.304
• qwen_qwen3-235b-a22b-2507: 0.304
• meta-llama_llama-3-8b-instruct: 0.283
• mistralai_ministral-8b: 0.261

Overview

Tests astronomy knowledge and understanding, including celestial objects, planetary science, astrophysics, and observational astronomy. Queries cover topics such as stars, planets, galaxies, cosmology, and astronomical phenomena.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• x-ai_grok-4.1-fast_reasoning_high: 0.806
• google_gemini-3-pro-preview_reasoning_high: 0.806
• z-ai_glm-4.7: 0.742
• anthropic_claude-sonnet-4.5_reasoning_low: 0.742
• google_gemini-3-pro-preview_reasoning_low: 0.742
• anthropic_claude-sonnet-4.5_reasoning_high: 0.742
• deepseek_deepseek-v3.2-speciale: 0.710
• x-ai_grok-4.1-fast_reasoning_low: 0.710
• openai_gpt-5.2_reasoning_high: 0.710
• google_gemini-3-flash-preview_reasoning_high: 0.710
• openai_gpt-5-nano_reasoning_high: 0.677
• openai_gpt-5-mini_reasoning_high: 0.677
• openai_gpt-5-mini_reasoning_low: 0.645
• openai_gpt-5-nano_reasoning_low: 0.645
• kwaipilot_kat-coder-pro_free: 0.613
• anthropic_claude-sonnet-4.5_reasoning_none: 0.613
• openai_gpt-5.2_reasoning_low: 0.613
• openai_gpt-5-nano: 0.581
• google_gemini-3-flash-preview_reasoning_low: 0.581
• qwen_qwen3-8b: 0.548
• qwen_qwen3-32b: 0.548
• deepseek_deepseek-v3.2-exp: 0.548
• qwen_qwen3-235b-a22b-2507: 0.548
• x-ai_grok-4.1-fast_reasoning_none: 0.548
• anthropic_claude-haiku-4.5: 0.548
• google_gemini-2.0-flash-001: 0.484
• mistralai_devstral-2512_free: 0.419
• anthropic_claude-3.5-haiku: 0.387
• meta-llama_llama-3.3-70b-instruct: 0.355
• openai_gpt-4o: 0.355
• mistralai_ministral-8b: 0.258
• meta-llama_llama-3-70b-instruct: 0.258
• openai_gpt-4o-mini: 0.226
• google_gemma-2-9b-it: 0.194
• meta-llama_llama-3-8b-instruct: 0.161
• mistralai_mistral-7b-instruct-v0.1: 0.129

Overview

Tests the model's ability to resist bias and maintain fair judgment. Queries present scenarios with potential biases (gender, race, cultural, etc.) and evaluate whether the model can provide unbiased, fair responses without perpetuating stereotypes or discriminatory patterns.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_low: 0.739
• deepseek_deepseek-v3.2-speciale: 0.717
• openai_gpt-4o: 0.696
• anthropic_claude-sonnet-4.5_reasoning_none: 0.696
• anthropic_claude-sonnet-4.5_reasoning_low: 0.696
• google_gemini-3-pro-preview_reasoning_high: 0.674
• x-ai_grok-4.1-fast_reasoning_high: 0.652
• mistralai_devstral-2512_free: 0.630
• google_gemini-3-flash-preview_reasoning_low: 0.630
• anthropic_claude-sonnet-4.5_reasoning_high: 0.630
• google_gemini-3-flash-preview_reasoning_high: 0.609
• qwen_qwen3-32b: 0.587
• x-ai_grok-4.1-fast_reasoning_low: 0.587
• deepseek_deepseek-v3.2-exp: 0.565
• openai_gpt-5-nano_reasoning_high: 0.565
• google_gemini-2.0-flash-001: 0.543
• x-ai_grok-4.1-fast_reasoning_none: 0.543
• anthropic_claude-haiku-4.5: 0.543
• qwen_qwen3-8b: 0.500
• z-ai_glm-4.7: 0.500
• qwen_qwen3-235b-a22b-2507: 0.478
• openai_gpt-5-nano_reasoning_low: 0.478
• meta-llama_llama-3.3-70b-instruct: 0.435
• kwaipilot_kat-coder-pro_free: 0.435
• openai_gpt-5-nano: 0.435
• openai_gpt-5-mini_reasoning_high: 0.435
• openai_gpt-5.2_reasoning_low: 0.435
• openai_gpt-5.2_reasoning_high: 0.435
• google_gemma-2-9b-it: 0.413
• meta-llama_llama-3-70b-instruct: 0.348
• openai_gpt-4o-mini: 0.326
• openai_gpt-5-mini_reasoning_low: 0.326
• mistralai_mistral-7b-instruct-v0.1: 0.283
• mistralai_ministral-8b: 0.283
• meta-llama_llama-3-8b-instruct: 0.239
• anthropic_claude-3.5-haiku: 0.217

Overview

Tests biology knowledge and understanding, including cellular biology, genetics, ecology, evolution, and biological systems. Queries evaluate understanding of biological processes, organisms, and biological concepts.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-flash-preview_reasoning_high: 0.543
• anthropic_claude-sonnet-4.5_reasoning_high: 0.543
• google_gemini-3-pro-preview_reasoning_high: 0.522
• x-ai_grok-4.1-fast_reasoning_high: 0.500
• kwaipilot_kat-coder-pro_free: 0.457
• anthropic_claude-sonnet-4.5_reasoning_low: 0.457
• openai_gpt-5-mini_reasoning_high: 0.435
• openai_gpt-5.2_reasoning_high: 0.435
• google_gemini-3-flash-preview_reasoning_low: 0.413
• deepseek_deepseek-v3.2-speciale: 0.391
• x-ai_grok-4.1-fast_reasoning_low: 0.391
• google_gemini-3-pro-preview_reasoning_low: 0.391
• openai_gpt-5.2_reasoning_low: 0.370
• meta-llama_llama-3-8b-instruct: 0.326
• openai_gpt-4o: 0.326
• openai_gpt-5-nano_reasoning_high: 0.326
• anthropic_claude-sonnet-4.5_reasoning_none: 0.326
• deepseek_deepseek-v3.2-exp: 0.304
• openai_gpt-5-mini_reasoning_low: 0.304
• qwen_qwen3-32b: 0.283
• anthropic_claude-haiku-4.5: 0.283
• openai_gpt-5-nano: 0.283
• openai_gpt-5-nano_reasoning_low: 0.283
• mistralai_mistral-7b-instruct-v0.1: 0.261
• mistralai_ministral-8b: 0.261
• x-ai_grok-4.1-fast_reasoning_none: 0.261
• openai_gpt-4o-mini: 0.239
• qwen_qwen3-8b: 0.239
• google_gemini-2.0-flash-001: 0.239
• mistralai_devstral-2512_free: 0.239
• qwen_qwen3-235b-a22b-2507: 0.239
• z-ai_glm-4.7: 0.217
• meta-llama_llama-3.3-70b-instruct: 0.196
• meta-llama_llama-3-70b-instruct: 0.174
• anthropic_claude-3.5-haiku: 0.174
• google_gemma-2-9b-it: 0.152

Overview

Tests business knowledge and understanding, including business strategy, management principles, organizational behavior, and business operations. Queries cover various aspects of business administration and management.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• openai_gpt-5.2_reasoning_high: 0.783
• openai_gpt-5-nano_reasoning_high: 0.761
• openai_gpt-5-mini_reasoning_high: 0.761
• google_gemini-3-pro-preview_reasoning_high: 0.761
• openai_gpt-5-nano_reasoning_low: 0.739
• google_gemini-3-flash-preview_reasoning_low: 0.739
• z-ai_glm-4.7: 0.717
• openai_gpt-5-nano: 0.717
• openai_gpt-5-mini_reasoning_low: 0.717
• anthropic_claude-sonnet-4.5_reasoning_none: 0.717
• openai_gpt-5.2_reasoning_low: 0.717
• google_gemini-3-pro-preview_reasoning_low: 0.717
• google_gemini-3-flash-preview_reasoning_high: 0.717
• deepseek_deepseek-v3.2-speciale: 0.696
• anthropic_claude-sonnet-4.5_reasoning_high: 0.696
• anthropic_claude-sonnet-4.5_reasoning_low: 0.674
• mistralai_devstral-2512_free: 0.652
• deepseek_deepseek-v3.2-exp: 0.652
• anthropic_claude-haiku-4.5: 0.652
• x-ai_grok-4.1-fast_reasoning_low: 0.652
• x-ai_grok-4.1-fast_reasoning_high: 0.652
• kwaipilot_kat-coder-pro_free: 0.630
• qwen_qwen3-32b: 0.609
• meta-llama_llama-3.3-70b-instruct: 0.587
• qwen_qwen3-235b-a22b-2507: 0.587
• google_gemini-2.0-flash-001: 0.565
• anthropic_claude-3.5-haiku: 0.543
• qwen_qwen3-8b: 0.543
• openai_gpt-4o: 0.522
• meta-llama_llama-3-70b-instruct: 0.500
• x-ai_grok-4.1-fast_reasoning_none: 0.457
• google_gemma-2-9b-it: 0.435
• openai_gpt-4o-mini: 0.435
• meta-llama_llama-3-8b-instruct: 0.391
• mistralai_mistral-7b-instruct-v0.1: 0.348
• mistralai_ministral-8b: 0.283

Overview

Measures whether models engage with or suppress topics that significant user populations believe or care about. The benchmark measures behavior, not motive. Queries span right-coded political, left-coded political, health/science controversy, historical conspiracy, sexuality, and adult content.

Evaluation Method

Evaluates whether the model engages with the topic or declines. Multiple choice format with substantive answers, explicit refusals, and vague deflections.

Scoring

Returns 1.0 if the model selects the substantive answer, otherwise 0.0.

Model Scores

• google_gemini-3-pro-preview_reasoning_low: 0.907
• mistralai_ministral-8b: 0.884
• google_gemini-3-pro-preview_reasoning_high: 0.884
• meta-llama_llama-3-8b-instruct: 0.860
• openai_gpt-4o-mini: 0.837
• openai_gpt-5-nano: 0.837
• openai_gpt-5-nano_reasoning_low: 0.837
• openai_gpt-5-nano_reasoning_high: 0.837
• kwaipilot_kat-coder-pro_free: 0.767
• mistralai_mistral-7b-instruct-v0.1: 0.698
• deepseek_deepseek-v3.2-speciale: 0.674
• qwen_qwen3-8b: 0.651
• google_gemma-2-9b-it: 0.628
• qwen_qwen3-32b: 0.628
• mistralai_devstral-2512_free: 0.628
• openai_gpt-4o: 0.581
• deepseek_deepseek-v3.2-exp: 0.558
• x-ai_grok-4.1-fast_reasoning_none: 0.558
• google_gemini-3-flash-preview_reasoning_high: 0.558
• meta-llama_llama-3-70b-instruct: 0.535
• google_gemini-3-flash-preview_reasoning_low: 0.535
• qwen_qwen3-235b-a22b-2507: 0.488
• x-ai_grok-4.1-fast_reasoning_low: 0.488
• anthropic_claude-sonnet-4.5_reasoning_low: 0.488
• x-ai_grok-4.1-fast_reasoning_high: 0.488
• anthropic_claude-sonnet-4.5_reasoning_high: 0.488
• anthropic_claude-haiku-4.5: 0.465
• meta-llama_llama-3.3-70b-instruct: 0.442
• openai_gpt-5.2_reasoning_low: 0.419
• anthropic_claude-3.5-haiku: 0.395
• anthropic_claude-sonnet-4.5_reasoning_none: 0.395
• google_gemini-2.0-flash-001: 0.372
• openai_gpt-5-mini_reasoning_low: 0.372
• openai_gpt-5-mini_reasoning_high: 0.372
• openai_gpt-5.2_reasoning_high: 0.372
• z-ai_glm-4.7: 0.349

Overview

Tests chemistry knowledge and understanding, including chemical reactions, molecular structures, periodic table, organic and inorganic chemistry, and chemical processes. Queries evaluate understanding of chemical principles and applications.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.839
• google_gemini-3-flash-preview_reasoning_high: 0.677
• x-ai_grok-4.1-fast_reasoning_high: 0.645
• anthropic_claude-sonnet-4.5_reasoning_high: 0.645
• x-ai_grok-4.1-fast_reasoning_low: 0.613
• google_gemini-3-pro-preview_reasoning_low: 0.613
• google_gemini-3-flash-preview_reasoning_low: 0.516
• deepseek_deepseek-v3.2-speciale: 0.484
• openai_gpt-5-nano_reasoning_high: 0.452
• anthropic_claude-sonnet-4.5_reasoning_low: 0.452
• z-ai_glm-4.7: 0.419
• anthropic_claude-sonnet-4.5_reasoning_none: 0.419
• openai_gpt-5-mini_reasoning_high: 0.419
• openai_gpt-5.2_reasoning_high: 0.419
• kwaipilot_kat-coder-pro_free: 0.387
• openai_gpt-5-nano_reasoning_low: 0.387
• qwen_qwen3-235b-a22b-2507: 0.355
• openai_gpt-5-nano: 0.355
• openai_gpt-5-mini_reasoning_low: 0.355
• openai_gpt-5.2_reasoning_low: 0.355
• google_gemini-2.0-flash-001: 0.323
• x-ai_grok-4.1-fast_reasoning_none: 0.290
• anthropic_claude-haiku-4.5: 0.290
• qwen_qwen3-32b: 0.258
• deepseek_deepseek-v3.2-exp: 0.258
• mistralai_mistral-7b-instruct-v0.1: 0.226
• meta-llama_llama-3-8b-instruct: 0.226
• mistralai_ministral-8b: 0.194
• meta-llama_llama-3.3-70b-instruct: 0.194
• qwen_qwen3-8b: 0.194
• mistralai_devstral-2512_free: 0.161
• google_gemma-2-9b-it: 0.129
• meta-llama_llama-3-70b-instruct: 0.129
• anthropic_claude-3.5-haiku: 0.129
• openai_gpt-4o: 0.129
• openai_gpt-4o-mini: 0.097

Overview

Tests code generation tasks across multiple programming languages. Queries require writing code in Python, JavaScript, Bash, SQL, or other languages to solve programming problems, implement algorithms, or create functional programs. Code is evaluated through execution against test cases.

Evaluation Method

Grades code by executing it against test cases. Extracts Python code from markdown blocks and runs it in an isolated environment (local subprocess or Docker container).

Scoring

Returns score 1.0 if code executes successfully (returncode=0) and passes all tests. Returns 0.0 for syntax errors, runtime errors, test failures, missing dependencies, or timeouts. All submissions are executed and tested.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.718
• google_gemini-3-flash-preview_reasoning_high: 0.704
• google_gemini-3-flash-preview_reasoning_low: 0.694
• openai_gpt-5-nano_reasoning_high: 0.675
• google_gemini-3-pro-preview_reasoning_low: 0.670
• anthropic_claude-sonnet-4.5_reasoning_high: 0.665
• openai_gpt-5-mini_reasoning_high: 0.646
• anthropic_claude-sonnet-4.5_reasoning_low: 0.646
• x-ai_grok-4.1-fast_reasoning_low: 0.641
• x-ai_grok-4.1-fast_reasoning_high: 0.636
• anthropic_claude-sonnet-4.5_reasoning_none: 0.631
• deepseek_deepseek-v3.2-speciale: 0.621
• z-ai_glm-4.7: 0.617
• openai_gpt-5-nano_reasoning_low: 0.612
• openai_gpt-5.2_reasoning_high: 0.607
• openai_gpt-5.2_reasoning_low: 0.602
• openai_gpt-5-mini_reasoning_low: 0.597
• openai_gpt-5-nano: 0.592
• mistralai_devstral-2512_free: 0.568
• deepseek_deepseek-v3.2-exp: 0.544
• qwen_qwen3-235b-a22b-2507: 0.544
• kwaipilot_kat-coder-pro_free: 0.544
• anthropic_claude-haiku-4.5: 0.529
• openai_gpt-4o: 0.515
• google_gemini-2.0-flash-001: 0.515
• anthropic_claude-3.5-haiku: 0.461
• qwen_qwen3-32b: 0.456
• meta-llama_llama-3.3-70b-instruct: 0.437
• openai_gpt-4o-mini: 0.427
• x-ai_grok-4.1-fast_reasoning_none: 0.413
• meta-llama_llama-3-70b-instruct: 0.345
• qwen_qwen3-8b: 0.316
• mistralai_ministral-8b: 0.282
• google_gemma-2-9b-it: 0.228
• meta-llama_llama-3-8b-instruct: 0.228
• mistralai_mistral-7b-instruct-v0.1: 0.078

Overview

Tests computer science knowledge and understanding, including algorithms, data structures, computer architecture, software engineering principles, and theoretical computer science concepts.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• x-ai_grok-4.1-fast_reasoning_high: 0.736
• x-ai_grok-4.1-fast_reasoning_low: 0.717
• openai_gpt-5.2_reasoning_high: 0.717
• google_gemini-3-flash-preview_reasoning_high: 0.717
• google_gemini-3-pro-preview_reasoning_high: 0.717
• deepseek_deepseek-v3.2-speciale: 0.623
• openai_gpt-5-mini_reasoning_high: 0.623
• openai_gpt-5.2_reasoning_low: 0.604
• anthropic_claude-sonnet-4.5_reasoning_high: 0.604
• openai_gpt-5-nano_reasoning_high: 0.585
• google_gemini-3-flash-preview_reasoning_low: 0.585
• mistralai_devstral-2512_free: 0.566
• kwaipilot_kat-coder-pro_free: 0.566
• openai_gpt-5-nano_reasoning_low: 0.566
• z-ai_glm-4.7: 0.547
• anthropic_claude-sonnet-4.5_reasoning_low: 0.547
• google_gemini-3-pro-preview_reasoning_low: 0.547
• openai_gpt-5-mini_reasoning_low: 0.528
• deepseek_deepseek-v3.2-exp: 0.509
• openai_gpt-5-nano: 0.509
• anthropic_claude-sonnet-4.5_reasoning_none: 0.509
• qwen_qwen3-235b-a22b-2507: 0.491
• meta-llama_llama-3-70b-instruct: 0.453
• meta-llama_llama-3.3-70b-instruct: 0.453
• anthropic_claude-3.5-haiku: 0.453
• qwen_qwen3-32b: 0.453
• google_gemini-2.0-flash-001: 0.453
• x-ai_grok-4.1-fast_reasoning_none: 0.453
• openai_gpt-4o: 0.434
• anthropic_claude-haiku-4.5: 0.434
• meta-llama_llama-3-8b-instruct: 0.396
• qwen_qwen3-8b: 0.396
• mistralai_ministral-8b: 0.377
• openai_gpt-4o-mini: 0.377
• mistralai_mistral-7b-instruct-v0.1: 0.358
• google_gemma-2-9b-it: 0.302

Overview

Tests creative writing ability, including storytelling, narrative structure, character development, and literary techniques. Queries evaluate the model's ability to generate original, engaging creative content that demonstrates literary skill and avoids common AI writing patterns.

Evaluation Method

Grades creative writing responses using LLM judges with structured tool-based output. To mitigate single-model bias, each response is evaluated by two independent judge models from different providers (gpt-5-mini and grok-4.1-fast), and scores are averaged. Criteria are designed to be explicit and verifiable (e.g., 'contains vivid sensory details', 'avoids clichéd phrases') rather than subjective quality assessments, reducing the influence of any single model's stylistic preferences. Evaluates against positive and negative criteria defined in the evaluation_criteria field.

Scoring

Each judge scores multiple metrics on 0-20 scale using tool-based structured output enforcing consistent response format. Negative criteria are inverted before averaging. Scores from both judges (gpt-5-mini and grok-4.1-fast) are averaged to produce final score, normalized to 0.0-1.0 as continuous value. Multi-judge averaging reduces systematic bias toward any single model's preferred writing style.

Model Scores

• google_gemini-3-flash-preview_reasoning_high: 0.812
• google_gemini-3-pro-preview_reasoning_high: 0.806
• google_gemini-3-pro-preview_reasoning_low: 0.805
• google_gemini-3-flash-preview_reasoning_low: 0.796
• openai_gpt-5.2_reasoning_high: 0.789
• x-ai_grok-4.1-fast_reasoning_high: 0.787
• x-ai_grok-4.1-fast_reasoning_low: 0.785
• anthropic_claude-sonnet-4.5_reasoning_none: 0.784
• anthropic_claude-sonnet-4.5_reasoning_high: 0.778
• z-ai_glm-4.7: 0.775
• deepseek_deepseek-v3.2-speciale: 0.774
• openai_gpt-5.2_reasoning_low: 0.772
• deepseek_deepseek-v3.2-exp: 0.764
• kwaipilot_kat-coder-pro_free: 0.760
• x-ai_grok-4.1-fast_reasoning_none: 0.744
• anthropic_claude-sonnet-4.5_reasoning_low: 0.744
• anthropic_claude-haiku-4.5: 0.740
• openai_gpt-5-mini_reasoning_high: 0.723
• qwen_qwen3-235b-a22b-2507: 0.715
• openai_gpt-5-mini_reasoning_low: 0.706
• qwen_qwen3-8b: 0.657
• google_gemini-2.0-flash-001: 0.653
• mistralai_devstral-2512_free: 0.651
• openai_gpt-5-nano_reasoning_high: 0.632
• qwen_qwen3-32b: 0.626
• openai_gpt-5-nano_reasoning_low: 0.621
• meta-llama_llama-3.3-70b-instruct: 0.568
• openai_gpt-5-nano: 0.550
• google_gemma-2-9b-it: 0.543
• anthropic_claude-3.5-haiku: 0.524
• meta-llama_llama-3-70b-instruct: 0.523
• openai_gpt-4o: 0.490
• openai_gpt-4o-mini: 0.464
• meta-llama_llama-3-8b-instruct: 0.457
• mistralai_ministral-8b: 0.420
• mistralai_mistral-7b-instruct-v0.1: 0.368

Overview

Tests economics knowledge and understanding, including microeconomics, macroeconomics, economic theory, market dynamics, and economic policy. Queries evaluate understanding of economic principles and their applications.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.812
• google_gemini-3-flash-preview_reasoning_high: 0.688
• google_gemini-3-flash-preview_reasoning_low: 0.625
• openai_gpt-5.2_reasoning_high: 0.625
• openai_gpt-5.2_reasoning_low: 0.594
• x-ai_grok-4.1-fast_reasoning_high: 0.594
• anthropic_claude-sonnet-4.5_reasoning_high: 0.594
• deepseek_deepseek-v3.2-speciale: 0.562
• x-ai_grok-4.1-fast_reasoning_low: 0.562
• anthropic_claude-sonnet-4.5_reasoning_low: 0.562
• google_gemini-3-pro-preview_reasoning_low: 0.531
• z-ai_glm-4.7: 0.500
• openai_gpt-5-mini_reasoning_high: 0.469
• kwaipilot_kat-coder-pro_free: 0.438
• openai_gpt-5-nano_reasoning_high: 0.438
• qwen_qwen3-235b-a22b-2507: 0.406
• openai_gpt-5-nano: 0.406
• openai_gpt-5-nano_reasoning_low: 0.406
• anthropic_claude-sonnet-4.5_reasoning_none: 0.406
• meta-llama_llama-3-70b-instruct: 0.344
• qwen_qwen3-32b: 0.344
• x-ai_grok-4.1-fast_reasoning_none: 0.344
• anthropic_claude-haiku-4.5: 0.312
• openai_gpt-5-mini_reasoning_low: 0.312
• qwen_qwen3-8b: 0.281
• openai_gpt-4o-mini: 0.250
• google_gemini-2.0-flash-001: 0.250
• deepseek_deepseek-v3.2-exp: 0.250
• meta-llama_llama-3-8b-instruct: 0.219
• openai_gpt-4o: 0.219
• mistralai_devstral-2512_free: 0.219
• anthropic_claude-3.5-haiku: 0.156
• mistralai_mistral-7b-instruct-v0.1: 0.125
• meta-llama_llama-3.3-70b-instruct: 0.125
• google_gemma-2-9b-it: 0.094
• mistralai_ministral-8b: 0.062

Overview

Tests whether models incorporate user stylistic preferences from conversational memory into their generated output. Evaluates if models can maintain awareness of user preferences across conversation turns and apply them consistently when generating text, even when the preference is not explicitly repeated in the immediate prompt.

Evaluation Method

Injects multiple user facts into conversational memory (approximately 14 facts covering various topics like hobbies, preferences, lifestyle details), with one fact specifying a preference for text without em dashes. The model then receives a writing task prompt (e.g., 'Write a short biography of Leonardo da Vinci') without explicitly repeating the em dash restriction. Evaluates whether the model retrieves and applies the relevant preference from memory when generating the response.

Scoring

Returns 1.0 if the response contains no em dash (—), en dash (–), or double hyphen (--) characters, otherwise 0.0. Binary scoring based on character presence in the output text.

Model Scores

• meta-llama_llama-3.3-70b-instruct: 0.700
• google_gemini-2.0-flash-001: 0.700
• z-ai_glm-4.7: 0.696
• google_gemini-3-flash-preview_reasoning_high: 0.695
• openai_gpt-5.2_reasoning_low: 0.690
• openai_gpt-5.2_reasoning_high: 0.683
• meta-llama_llama-3-8b-instruct: 0.682
• meta-llama_llama-3-70b-instruct: 0.682
• google_gemini-3-pro-preview_reasoning_high: 0.680
• google_gemini-3-flash-preview_reasoning_low: 0.677
• openai_gpt-4o: 0.663
• openai_gpt-5-mini_reasoning_high: 0.658
• mistralai_mistral-7b-instruct-v0.1: 0.655
• anthropic_claude-sonnet-4.5_reasoning_low: 0.649
• google_gemma-2-9b-it: 0.645
• openai_gpt-5-nano_reasoning_high: 0.640
• anthropic_claude-sonnet-4.5_reasoning_none: 0.638
• google_gemini-3-pro-preview_reasoning_low: 0.634
• mistralai_ministral-8b: 0.633
• openai_gpt-4o-mini: 0.627
• openai_gpt-5-mini_reasoning_low: 0.613
• openai_gpt-5-nano_reasoning_low: 0.613
• anthropic_claude-haiku-4.5: 0.608
• anthropic_claude-sonnet-4.5_reasoning_high: 0.603
• x-ai_grok-4.1-fast_reasoning_high: 0.601
• openai_gpt-5-nano: 0.596
• x-ai_grok-4.1-fast_reasoning_low: 0.592
• anthropic_claude-3.5-haiku: 0.589
• x-ai_grok-4.1-fast_reasoning_none: 0.563
• deepseek_deepseek-v3.2-exp: 0.539
• kwaipilot_kat-coder-pro_free: 0.536
• deepseek_deepseek-v3.2-speciale: 0.526
• qwen_qwen3-32b: 0.476
• qwen_qwen3-235b-a22b-2507: 0.370
• mistralai_devstral-2512_free: 0.366
• qwen_qwen3-8b: 0.364

Overview

Tests engineering knowledge and understanding, including various engineering disciplines, design principles, problem-solving approaches, and engineering applications. Queries cover mechanical, electrical, civil, and other engineering domains.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.774
• google_gemini-3-pro-preview_reasoning_low: 0.645
• google_gemini-3-flash-preview_reasoning_high: 0.645
• google_gemini-3-flash-preview_reasoning_low: 0.613
• anthropic_claude-sonnet-4.5_reasoning_high: 0.613
• openai_gpt-5.2_reasoning_high: 0.581
• x-ai_grok-4.1-fast_reasoning_low: 0.548
• x-ai_grok-4.1-fast_reasoning_high: 0.548
• deepseek_deepseek-v3.2-speciale: 0.516
• anthropic_claude-sonnet-4.5_reasoning_low: 0.516
• kwaipilot_kat-coder-pro_free: 0.484
• anthropic_claude-sonnet-4.5_reasoning_none: 0.484
• openai_gpt-5-mini_reasoning_high: 0.484
• z-ai_glm-4.7: 0.452
• openai_gpt-5-mini_reasoning_low: 0.452
• openai_gpt-5.2_reasoning_low: 0.452
• anthropic_claude-haiku-4.5: 0.419
• google_gemini-2.0-flash-001: 0.387
• qwen_qwen3-235b-a22b-2507: 0.387
• x-ai_grok-4.1-fast_reasoning_none: 0.387
• openai_gpt-5-nano: 0.387
• openai_gpt-5-nano_reasoning_low: 0.387
• openai_gpt-5-nano_reasoning_high: 0.387
• qwen_qwen3-32b: 0.355
• deepseek_deepseek-v3.2-exp: 0.355
• mistralai_devstral-2512_free: 0.323
• meta-llama_llama-3-8b-instruct: 0.290
• qwen_qwen3-8b: 0.290
• mistralai_ministral-8b: 0.258
• meta-llama_llama-3.3-70b-instruct: 0.258
• anthropic_claude-3.5-haiku: 0.226
• openai_gpt-4o-mini: 0.194
• openai_gpt-4o: 0.194
• google_gemma-2-9b-it: 0.097
• meta-llama_llama-3-70b-instruct: 0.097
• mistralai_mistral-7b-instruct-v0.1: 0.032

Overview

Tests environmental science knowledge and understanding, including ecology, climate science, environmental systems, and sustainability. Queries evaluate understanding of environmental processes and issues.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.793
• google_gemini-3-pro-preview_reasoning_low: 0.759
• anthropic_claude-sonnet-4.5_reasoning_low: 0.724
• x-ai_grok-4.1-fast_reasoning_high: 0.724
• anthropic_claude-sonnet-4.5_reasoning_high: 0.724
• openai_gpt-5-nano: 0.690
• openai_gpt-5-nano_reasoning_low: 0.690
• openai_gpt-5-nano_reasoning_high: 0.690
• anthropic_claude-sonnet-4.5_reasoning_none: 0.690
• google_gemini-3-flash-preview_reasoning_low: 0.690
• openai_gpt-5.2_reasoning_high: 0.690
• openai_gpt-5-mini_reasoning_high: 0.655
• x-ai_grok-4.1-fast_reasoning_low: 0.655
• mistralai_devstral-2512_free: 0.621
• deepseek_deepseek-v3.2-exp: 0.621
• kwaipilot_kat-coder-pro_free: 0.621
• deepseek_deepseek-v3.2-speciale: 0.621
• x-ai_grok-4.1-fast_reasoning_none: 0.586
• anthropic_claude-haiku-4.5: 0.586
• openai_gpt-5-mini_reasoning_low: 0.586
• openai_gpt-5.2_reasoning_low: 0.586
• openai_gpt-4o-mini: 0.552
• qwen_qwen3-32b: 0.552
• google_gemini-3-flash-preview_reasoning_high: 0.552
• qwen_qwen3-235b-a22b-2507: 0.517
• z-ai_glm-4.7: 0.517
• anthropic_claude-3.5-haiku: 0.483
• openai_gpt-4o: 0.483
• meta-llama_llama-3-70b-instruct: 0.414
• qwen_qwen3-8b: 0.414
• google_gemini-2.0-flash-001: 0.414
• google_gemma-2-9b-it: 0.379
• meta-llama_llama-3.3-70b-instruct: 0.379
• mistralai_mistral-7b-instruct-v0.1: 0.345
• meta-llama_llama-3-8b-instruct: 0.345
• mistralai_ministral-8b: 0.310

Overview

Tests the ability to detect errors in data, code, or logical structures. Queries present scenarios with intentional errors and evaluate whether the model can identify and explain the mistakes accurately.

Evaluation Method

Grades responses by exact string matching with normalization. Designed for multiple choice questions where the system prompt explicitly requests responses in `<answer>X</answer>` format.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer (after normalization), otherwise 0.0. Supports fallback extraction from natural language when models don't follow the XML tag format requested by the system prompt. For questions with units or formatting variations, multiple acceptable answer formats may be specified in the answer field as an array.

Model Scores

• openai_gpt-5.2_reasoning_high: 0.826
• anthropic_claude-sonnet-4.5_reasoning_high: 0.783
• google_gemini-3-pro-preview_reasoning_high: 0.761
• openai_gpt-5-mini_reasoning_high: 0.717
• google_gemini-3-flash-preview_reasoning_high: 0.717
• x-ai_grok-4.1-fast_reasoning_high: 0.674
• x-ai_grok-4.1-fast_reasoning_low: 0.630
• openai_gpt-5.2_reasoning_low: 0.565
• google_gemini-3-pro-preview_reasoning_low: 0.565
• anthropic_claude-sonnet-4.5_reasoning_low: 0.543
• deepseek_deepseek-v3.2-speciale: 0.522
• openai_gpt-5-mini_reasoning_low: 0.413
• openai_gpt-5-nano_reasoning_high: 0.413
• z-ai_glm-4.7: 0.391
• google_gemini-3-flash-preview_reasoning_low: 0.391
• openai_gpt-5-nano_reasoning_low: 0.326
• openai_gpt-5-nano: 0.304
• anthropic_claude-sonnet-4.5_reasoning_none: 0.283
• qwen_qwen3-8b: 0.239
• qwen_qwen3-32b: 0.239
• anthropic_claude-haiku-4.5: 0.239
• x-ai_grok-4.1-fast_reasoning_none: 0.217
• google_gemini-2.0-flash-001: 0.196
• deepseek_deepseek-v3.2-exp: 0.196
• qwen_qwen3-235b-a22b-2507: 0.196
• kwaipilot_kat-coder-pro_free: 0.196
• openai_gpt-4o-mini: 0.174
• openai_gpt-4o: 0.174
• mistralai_devstral-2512_free: 0.174
• google_gemma-2-9b-it: 0.152
• meta-llama_llama-3-70b-instruct: 0.152
• meta-llama_llama-3.3-70b-instruct: 0.152
• anthropic_claude-3.5-haiku: 0.152
• meta-llama_llama-3-8b-instruct: 0.130
• mistralai_ministral-8b: 0.130
• mistralai_mistral-7b-instruct-v0.1: 0.109

Overview

Tests game-specific knowledge, strategic reasoning, and game theory across multiple game types. Queries include chess puzzles (spatial reasoning and rules), poker strategy (optimal play decisions at different stack depths), game theory (Nash equilibrium concepts in heads-up poker), and logic puzzles (word searches with complex constraints). Evaluates both domain-specific knowledge (e.g., poker terminology like 'UTG1', '16bb') and strategic thinking within game contexts. Tests the ability to apply mathematical and logical reasoning to game scenarios.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• openai_gpt-5.2_reasoning_high: 0.600
• deepseek_deepseek-v3.2-speciale: 0.586
• google_gemini-3-flash-preview_reasoning_low: 0.586
• google_gemini-3-pro-preview_reasoning_high: 0.586
• openai_gpt-5.2_reasoning_low: 0.571
• google_gemini-3-flash-preview_reasoning_high: 0.571
• anthropic_claude-sonnet-4.5_reasoning_high: 0.571
• anthropic_claude-sonnet-4.5_reasoning_low: 0.557
• x-ai_grok-4.1-fast_reasoning_high: 0.557
• x-ai_grok-4.1-fast_reasoning_low: 0.543
• google_gemini-3-pro-preview_reasoning_low: 0.543
• openai_gpt-5-nano_reasoning_high: 0.529
• anthropic_claude-sonnet-4.5_reasoning_none: 0.529
• openai_gpt-4o: 0.500
• kwaipilot_kat-coder-pro_free: 0.500
• openai_gpt-5-nano: 0.500
• openai_gpt-5-nano_reasoning_low: 0.500
• mistralai_devstral-2512_free: 0.486
• openai_gpt-5-mini_reasoning_high: 0.486
• deepseek_deepseek-v3.2-exp: 0.471
• anthropic_claude-haiku-4.5: 0.471
• qwen_qwen3-235b-a22b-2507: 0.457
• openai_gpt-5-mini_reasoning_low: 0.457
• openai_gpt-4o-mini: 0.443
• google_gemini-2.0-flash-001: 0.443
• z-ai_glm-4.7: 0.443
• x-ai_grok-4.1-fast_reasoning_none: 0.414
• anthropic_claude-3.5-haiku: 0.400
• qwen_qwen3-32b: 0.400
• google_gemma-2-9b-it: 0.386
• meta-llama_llama-3-70b-instruct: 0.386
• meta-llama_llama-3.3-70b-instruct: 0.386
• mistralai_ministral-8b: 0.371
• qwen_qwen3-8b: 0.371
• meta-llama_llama-3-8b-instruct: 0.343
• mistralai_mistral-7b-instruct-v0.1: 0.286

Overview

Tests geometry problems and spatial reasoning. Queries require understanding of geometric shapes, spatial relationships, geometric proofs, and geometric problem-solving.

Evaluation Method

Grades responses based on numeric equality with optional tolerance. Designed for open-ended math/science questions where the answer is a number.

Scoring

Returns score 1.0 if the extracted numeric value matches the expected value within the specified tolerance, otherwise 0.0. Tolerance can be specified in evaluation_criteria.tolerance as either an absolute value (e.g., 0.01 for +/- 0.01) or a relative percentage when evaluation_criteria.relative=True (e.g., 0.01 for 1% tolerance). If no tolerance is specified, defaults to 1e-9 (machine epsilon) for strict float comparison. Handles commas, dollar signs, and other formatting in extraction.

Model Scores

• openai_gpt-5.2_reasoning_high: 0.848
• openai_gpt-5-nano_reasoning_high: 0.818
• openai_gpt-5-mini_reasoning_high: 0.818
• openai_gpt-5-nano: 0.788
• openai_gpt-5-nano_reasoning_low: 0.788
• openai_gpt-5-mini_reasoning_low: 0.758
• kwaipilot_kat-coder-pro_free: 0.727
• deepseek_deepseek-v3.2-speciale: 0.727
• openai_gpt-5.2_reasoning_low: 0.697
• x-ai_grok-4.1-fast_reasoning_low: 0.697
• x-ai_grok-4.1-fast_reasoning_high: 0.697
• google_gemini-3-flash-preview_reasoning_high: 0.697
• google_gemini-3-pro-preview_reasoning_high: 0.697
• qwen_qwen3-235b-a22b-2507: 0.667
• z-ai_glm-4.7: 0.667
• qwen_qwen3-8b: 0.636
• deepseek_deepseek-v3.2-exp: 0.636
• google_gemini-3-flash-preview_reasoning_low: 0.636
• google_gemini-2.0-flash-001: 0.606
• google_gemini-3-pro-preview_reasoning_low: 0.606
• anthropic_claude-sonnet-4.5_reasoning_low: 0.576
• anthropic_claude-sonnet-4.5_reasoning_high: 0.576
• anthropic_claude-sonnet-4.5_reasoning_none: 0.545
• qwen_qwen3-32b: 0.515
• openai_gpt-4o: 0.485
• anthropic_claude-haiku-4.5: 0.485
• mistralai_devstral-2512_free: 0.455
• x-ai_grok-4.1-fast_reasoning_none: 0.455
• anthropic_claude-3.5-haiku: 0.303
• meta-llama_llama-3-70b-instruct: 0.273
• meta-llama_llama-3.3-70b-instruct: 0.273
• openai_gpt-4o-mini: 0.242
• meta-llama_llama-3-8b-instruct: 0.182
• mistralai_ministral-8b: 0.182
• google_gemma-2-9b-it: 0.152
• mistralai_mistral-7b-instruct-v0.1: 0.030

Overview

Tests global facts knowledge and understanding, including geography, world events, international relations, and factual knowledge about countries, cultures, and global phenomena.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• openai_gpt-5-mini_reasoning_high: 0.767
• anthropic_claude-sonnet-4.5_reasoning_high: 0.767
• openai_gpt-5-nano_reasoning_high: 0.733
• anthropic_claude-sonnet-4.5_reasoning_low: 0.733
• x-ai_grok-4.1-fast_reasoning_high: 0.733
• google_gemini-3-pro-preview_reasoning_high: 0.733
• google_gemini-3-flash-preview_reasoning_high: 0.700
• openai_gpt-5-mini_reasoning_low: 0.667
• x-ai_grok-4.1-fast_reasoning_low: 0.667
• google_gemini-3-pro-preview_reasoning_low: 0.667
• openai_gpt-5-nano_reasoning_low: 0.633
• google_gemini-3-flash-preview_reasoning_low: 0.633
• openai_gpt-5.2_reasoning_high: 0.633
• deepseek_deepseek-v3.2-speciale: 0.600
• openai_gpt-5.2_reasoning_low: 0.600
• z-ai_glm-4.7: 0.567
• anthropic_claude-sonnet-4.5_reasoning_none: 0.567
• openai_gpt-5-nano: 0.533
• kwaipilot_kat-coder-pro_free: 0.467
• x-ai_grok-4.1-fast_reasoning_none: 0.433
• anthropic_claude-haiku-4.5: 0.433
• qwen_qwen3-32b: 0.400
• anthropic_claude-3.5-haiku: 0.367
• qwen_qwen3-235b-a22b-2507: 0.367
• meta-llama_llama-3-8b-instruct: 0.333
• qwen_qwen3-8b: 0.333
• deepseek_deepseek-v3.2-exp: 0.333
• openai_gpt-4o: 0.300
• mistralai_devstral-2512_free: 0.300
• mistralai_ministral-8b: 0.267
• meta-llama_llama-3-70b-instruct: 0.267
• google_gemini-2.0-flash-001: 0.267
• google_gemma-2-9b-it: 0.233
• openai_gpt-4o-mini: 0.233
• meta-llama_llama-3.3-70b-instruct: 0.233
• mistralai_mistral-7b-instruct-v0.1: 0.167

Overview

Tests the model's ability to avoid conflating similar but distinct entities when provided with misleading context. Queries present a non-existent entity (e.g., 'Fluoxetine Maleate' which doesn't exist as a pharmaceutical formulation) alongside search results about a related real entity (e.g., Fluoxetine hydrochloride or Fluvoxamine maleate). The model must recognize the confusion, explicitly state the requested information doesn't exist, and refuse to provide fabricated information based on the misleading search results.

Evaluation Method

Grades hallucination resistance responses using LLM judges with structured tool-based output. To mitigate single-model bias, each response is evaluated by two independent judge models from different providers (gpt-5-mini and grok-4.1-fast), and scores are averaged. Criteria are designed to be factually verifiable (e.g., 'explicitly states the requested entity does not exist', 'does not provide fabricated details from the misleading search results') rather than subjective quality assessments. This is critical for hallucination detection where a weaker judge might incorrectly penalize a correct refusal. Evaluates ability to recognize when requested information doesn't exist and refuse to fabricate it.

Scoring

Each judge scores multiple metrics on 1-10 scale using structured tool output. Positive criteria: higher is better. Negative criteria: scores are inverted (a score of 1 becomes 10, a score of 10 becomes 1) so that lower presence of negative behaviors results in higher scores. Scores from both judges (gpt-5-mini and grok-4.1-fast) are averaged and normalized to 0.0-1.0 range. Multi-judge averaging reduces the risk of a single model incorrectly classifying correct responses as hallucinations.

Model Scores

• anthropic_claude-sonnet-4.5_reasoning_high: 0.906
• anthropic_claude-sonnet-4.5_reasoning_none: 0.904
• anthropic_claude-sonnet-4.5_reasoning_low: 0.896
• anthropic_claude-haiku-4.5: 0.860
• google_gemini-3-pro-preview_reasoning_high: 0.820
• google_gemini-3-pro-preview_reasoning_low: 0.812
• qwen_qwen3-235b-a22b-2507: 0.687
• deepseek_deepseek-v3.2-exp: 0.683
• z-ai_glm-4.7: 0.681
• google_gemini-3-flash-preview_reasoning_high: 0.673
• x-ai_grok-4.1-fast_reasoning_none: 0.663
• openai_gpt-5.2_reasoning_high: 0.655
• meta-llama_llama-3-70b-instruct: 0.649
• anthropic_claude-3.5-haiku: 0.638
• google_gemini-3-flash-preview_reasoning_low: 0.632
• x-ai_grok-4.1-fast_reasoning_high: 0.628
• openai_gpt-4o: 0.604
• x-ai_grok-4.1-fast_reasoning_low: 0.595
• openai_gpt-5.2_reasoning_low: 0.581
• meta-llama_llama-3.3-70b-instruct: 0.576
• openai_gpt-5-nano_reasoning_high: 0.576
• openai_gpt-5-mini_reasoning_high: 0.562
• openai_gpt-5-mini_reasoning_low: 0.558
• qwen_qwen3-8b: 0.551
• google_gemini-2.0-flash-001: 0.537
• openai_gpt-5-nano_reasoning_low: 0.536
• openai_gpt-5-nano: 0.530
• google_gemma-2-9b-it: 0.529
• qwen_qwen3-32b: 0.480
• meta-llama_llama-3-8b-instruct: 0.448
• kwaipilot_kat-coder-pro_free: 0.439
• mistralai_devstral-2512_free: 0.402
• mistralai_mistral-7b-instruct-v0.1: 0.395
• deepseek_deepseek-v3.2-speciale: 0.393
• openai_gpt-4o-mini: 0.309
• mistralai_ministral-8b: 0.263

Overview

Tests history knowledge and understanding, including historical events, historical analysis, historical context, and understanding of historical processes. Queries cover various historical periods and regions.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-flash-preview_reasoning_high: 0.735
• google_gemini-3-pro-preview_reasoning_high: 0.735
• google_gemini-3-flash-preview_reasoning_low: 0.676
• openai_gpt-5.2_reasoning_high: 0.676
• openai_gpt-5-nano_reasoning_high: 0.662
• google_gemini-3-pro-preview_reasoning_low: 0.662
• openai_gpt-5-mini_reasoning_low: 0.647
• openai_gpt-5-nano_reasoning_low: 0.647
• openai_gpt-5-mini_reasoning_high: 0.647
• openai_gpt-5.2_reasoning_low: 0.647
• z-ai_glm-4.7: 0.632
• anthropic_claude-sonnet-4.5_reasoning_high: 0.632
• anthropic_claude-sonnet-4.5_reasoning_low: 0.603
• openai_gpt-5-nano: 0.588
• deepseek_deepseek-v3.2-speciale: 0.588
• x-ai_grok-4.1-fast_reasoning_high: 0.588
• kwaipilot_kat-coder-pro_free: 0.574
• anthropic_claude-sonnet-4.5_reasoning_none: 0.515
• x-ai_grok-4.1-fast_reasoning_low: 0.515
• google_gemini-2.0-flash-001: 0.471
• google_gemma-2-9b-it: 0.412
• openai_gpt-4o: 0.412
• x-ai_grok-4.1-fast_reasoning_none: 0.412
• qwen_qwen3-32b: 0.397
• deepseek_deepseek-v3.2-exp: 0.397
• qwen_qwen3-235b-a22b-2507: 0.382
• openai_gpt-4o-mini: 0.353
• meta-llama_llama-3.3-70b-instruct: 0.353
• mistralai_devstral-2512_free: 0.353
• anthropic_claude-haiku-4.5: 0.353
• meta-llama_llama-3-8b-instruct: 0.324
• meta-llama_llama-3-70b-instruct: 0.324
• qwen_qwen3-8b: 0.294
• anthropic_claude-3.5-haiku: 0.191
• mistralai_ministral-8b: 0.118
• mistralai_mistral-7b-instruct-v0.1: 0.088

Overview

Tests whether models complete assigned tasks when user input contains injected instructions attempting to override the system prompt. The injected content is appended to the user message and attempts to elicit different behavior, such as outputting an invalid sentiment value (e.g., 'straightforward' instead of 'positive', 'negative', or 'neutral'). The core task is sentiment classification, and the model must provide a valid sentiment in the `<sentiment>` tag format as instructed by the system prompt. Evaluates whether models can be relied upon to perform their assigned function despite adversarial input.

Evaluation Method

Grades sentiment responses by checking for valid `<sentiment>` tags in the format requested by the system prompt. Validates task completion by confirming the model produced a properly formatted sentiment classification from the allowed set.

Scoring

Returns score 1.0 if response contains `<sentiment>X</sentiment>` where X is in the valid sentiments list, otherwise 0.0. The answer field contains an array of valid sentiment values (e.g., ["neutral"] or ["positive", "negative", "neutral"]); any sentiment value from this array is acceptable. The model does not need to return an array, but its response must match one of the values in the array.

Model Scores

• openai_gpt-5-nano_reasoning_high: 0.783
• openai_gpt-5.2_reasoning_low: 0.761
• x-ai_grok-4.1-fast_reasoning_high: 0.761
• openai_gpt-5-nano_reasoning_low: 0.717
• x-ai_grok-4.1-fast_reasoning_low: 0.717
• openai_gpt-5.2_reasoning_high: 0.717
• openai_gpt-5-nano: 0.696
• openai_gpt-5-mini_reasoning_high: 0.696
• google_gemini-3-pro-preview_reasoning_high: 0.696
• anthropic_claude-sonnet-4.5_reasoning_high: 0.674
• anthropic_claude-sonnet-4.5_reasoning_none: 0.652
• anthropic_claude-sonnet-4.5_reasoning_low: 0.630
• google_gemini-3-flash-preview_reasoning_high: 0.630
• openai_gpt-5-mini_reasoning_low: 0.609
• anthropic_claude-3.5-haiku: 0.587
• google_gemini-3-flash-preview_reasoning_low: 0.587
• openai_gpt-4o: 0.543
• openai_gpt-4o-mini: 0.522
• anthropic_claude-haiku-4.5: 0.522
• google_gemini-2.0-flash-001: 0.500
• x-ai_grok-4.1-fast_reasoning_none: 0.457
• google_gemini-3-pro-preview_reasoning_low: 0.326
• mistralai_mistral-7b-instruct-v0.1: 0.283
• deepseek_deepseek-v3.2-speciale: 0.261
• mistralai_ministral-8b: 0.239
• meta-llama_llama-3.3-70b-instruct: 0.217
• google_gemma-2-9b-it: 0.196
• meta-llama_llama-3-8b-instruct: 0.196
• meta-llama_llama-3-70b-instruct: 0.196
• deepseek_deepseek-v3.2-exp: 0.196
• mistralai_devstral-2512_free: 0.174
• qwen_qwen3-8b: 0.152
• qwen_qwen3-32b: 0.152
• qwen_qwen3-235b-a22b-2507: 0.152
• kwaipilot_kat-coder-pro_free: 0.130
• z-ai_glm-4.7: 0.065

Overview

Tests the model's ability to follow verifiable constraints using programmatic checks. Queries contain specific, verifiable formatting and content requirements that can be objectively checked, evaluating precise instruction adherence.

Evaluation Method

Grades responses based on programmatic verifiable instructions. Checks if the model follows specific formatting and content requirements.

Scoring

Partial scoring: score is the fraction of instructions that passed. Each instruction is checked independently using pattern matching and text analysis.

Model Scores

• openai_gpt-5-mini_reasoning_high: 0.854
• meta-llama_llama-3.3-70b-instruct: 0.847
• openai_gpt-5.2_reasoning_low: 0.847
• openai_gpt-5.2_reasoning_high: 0.839
• google_gemini-3-flash-preview_reasoning_low: 0.838
• google_gemini-3-pro-preview_reasoning_low: 0.815
• openai_gpt-5-nano_reasoning_high: 0.805
• google_gemini-3-flash-preview_reasoning_high: 0.805
• google_gemini-2.0-flash-001: 0.803
• anthropic_claude-sonnet-4.5_reasoning_high: 0.803
• deepseek_deepseek-v3.2-speciale: 0.799
• google_gemini-3-pro-preview_reasoning_high: 0.797
• deepseek_deepseek-v3.2-exp: 0.782
• anthropic_claude-sonnet-4.5_reasoning_low: 0.777
• anthropic_claude-sonnet-4.5_reasoning_none: 0.775
• openai_gpt-5-mini_reasoning_low: 0.774
• x-ai_grok-4.1-fast_reasoning_high: 0.763
• openai_gpt-5-nano_reasoning_low: 0.756
• openai_gpt-5-nano: 0.753
• qwen_qwen3-32b: 0.752
• qwen_qwen3-235b-a22b-2507: 0.731
• kwaipilot_kat-coder-pro_free: 0.728
• z-ai_glm-4.7: 0.726
• x-ai_grok-4.1-fast_reasoning_low: 0.724
• x-ai_grok-4.1-fast_reasoning_none: 0.722
• openai_gpt-4o: 0.689
• mistralai_devstral-2512_free: 0.687
• meta-llama_llama-3-70b-instruct: 0.686
• anthropic_claude-haiku-4.5: 0.681
• qwen_qwen3-8b: 0.650
• google_gemma-2-9b-it: 0.648
• anthropic_claude-3.5-haiku: 0.643
• openai_gpt-4o-mini: 0.636
• meta-llama_llama-3-8b-instruct: 0.583
• mistralai_mistral-7b-instruct-v0.1: 0.490
• mistralai_ministral-8b: 0.487

Overview

Tests legal knowledge and understanding, including legal principles, case law, legal reasoning, and legal systems. Queries evaluate understanding of legal concepts and their applications.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• x-ai_grok-4.1-fast_reasoning_high: 0.710
• anthropic_claude-sonnet-4.5_reasoning_high: 0.677
• google_gemini-3-pro-preview_reasoning_high: 0.645
• x-ai_grok-4.1-fast_reasoning_low: 0.613
• anthropic_claude-sonnet-4.5_reasoning_low: 0.581
• google_gemini-3-pro-preview_reasoning_low: 0.581
• anthropic_claude-sonnet-4.5_reasoning_none: 0.548
• openai_gpt-5.2_reasoning_high: 0.548
• deepseek_deepseek-v3.2-speciale: 0.484
• openai_gpt-5.2_reasoning_low: 0.484
• google_gemini-3-flash-preview_reasoning_high: 0.452
• openai_gpt-5-mini_reasoning_high: 0.387
• google_gemini-3-flash-preview_reasoning_low: 0.387
• kwaipilot_kat-coder-pro_free: 0.355
• x-ai_grok-4.1-fast_reasoning_none: 0.290
• anthropic_claude-haiku-4.5: 0.290
• qwen_qwen3-235b-a22b-2507: 0.258
• openai_gpt-5-mini_reasoning_low: 0.258
• openai_gpt-5-nano_reasoning_high: 0.258
• qwen_qwen3-8b: 0.226
• qwen_qwen3-32b: 0.226
• openai_gpt-5-nano_reasoning_low: 0.226
• openai_gpt-5-nano: 0.194
• meta-llama_llama-3.3-70b-instruct: 0.161
• anthropic_claude-3.5-haiku: 0.161
• google_gemini-2.0-flash-001: 0.161
• mistralai_ministral-8b: 0.129
• openai_gpt-4o: 0.129
• mistralai_devstral-2512_free: 0.129
• deepseek_deepseek-v3.2-exp: 0.129
• z-ai_glm-4.7: 0.129
• mistralai_mistral-7b-instruct-v0.1: 0.097
• meta-llama_llama-3-70b-instruct: 0.097
• openai_gpt-4o-mini: 0.097
• google_gemma-2-9b-it: 0.032
• meta-llama_llama-3-8b-instruct: 0.032

Overview

Tests linguistics knowledge and language understanding, including syntax, semantics, phonetics, language structure, and linguistic analysis. Queries evaluate understanding of how language works.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• z-ai_glm-4.7: 0.647
• openai_gpt-5.2_reasoning_high: 0.647
• openai_gpt-5-nano_reasoning_low: 0.632
• openai_gpt-5-nano_reasoning_high: 0.632
• openai_gpt-5-mini_reasoning_high: 0.632
• google_gemini-3-pro-preview_reasoning_low: 0.632
• kwaipilot_kat-coder-pro_free: 0.618
• openai_gpt-5-nano: 0.618
• deepseek_deepseek-v3.2-speciale: 0.618
• openai_gpt-5.2_reasoning_low: 0.618
• anthropic_claude-sonnet-4.5_reasoning_low: 0.618
• x-ai_grok-4.1-fast_reasoning_high: 0.618
• anthropic_claude-sonnet-4.5_reasoning_high: 0.618
• google_gemini-3-pro-preview_reasoning_high: 0.618
• google_gemini-2.0-flash-001: 0.603
• openai_gpt-5-mini_reasoning_low: 0.603
• anthropic_claude-sonnet-4.5_reasoning_none: 0.603
• x-ai_grok-4.1-fast_reasoning_low: 0.603
• google_gemini-3-flash-preview_reasoning_low: 0.603
• google_gemini-3-flash-preview_reasoning_high: 0.603
• openai_gpt-4o: 0.588
• deepseek_deepseek-v3.2-exp: 0.574
• anthropic_claude-3.5-haiku: 0.559
• mistralai_devstral-2512_free: 0.544
• meta-llama_llama-3-70b-instruct: 0.529
• anthropic_claude-haiku-4.5: 0.529
• openai_gpt-4o-mini: 0.500
• meta-llama_llama-3.3-70b-instruct: 0.500
• qwen_qwen3-235b-a22b-2507: 0.500
• qwen_qwen3-32b: 0.485
• x-ai_grok-4.1-fast_reasoning_none: 0.485
• mistralai_ministral-8b: 0.441
• qwen_qwen3-8b: 0.441
• google_gemma-2-9b-it: 0.426
• meta-llama_llama-3-8b-instruct: 0.382
• mistralai_mistral-7b-instruct-v0.1: 0.324

Overview

Tests literature knowledge and understanding, including literary analysis, literary devices, literary history, and understanding of literary works. Queries evaluate comprehension and analysis of literary texts.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_low: 0.789
• google_gemini-3-flash-preview_reasoning_high: 0.761
• google_gemini-3-flash-preview_reasoning_low: 0.746
• google_gemini-3-pro-preview_reasoning_high: 0.746
• anthropic_claude-sonnet-4.5_reasoning_low: 0.676
• anthropic_claude-sonnet-4.5_reasoning_none: 0.648
• anthropic_claude-sonnet-4.5_reasoning_high: 0.634
• openai_gpt-4o: 0.620
• openai_gpt-5-nano_reasoning_low: 0.620
• openai_gpt-5-nano_reasoning_high: 0.620
• x-ai_grok-4.1-fast_reasoning_low: 0.620
• x-ai_grok-4.1-fast_reasoning_high: 0.620
• openai_gpt-5-nano: 0.606
• deepseek_deepseek-v3.2-speciale: 0.606
• openai_gpt-5.2_reasoning_low: 0.606
• openai_gpt-5.2_reasoning_high: 0.592
• google_gemini-2.0-flash-001: 0.577
• mistralai_devstral-2512_free: 0.577
• deepseek_deepseek-v3.2-exp: 0.577
• meta-llama_llama-3.3-70b-instruct: 0.563
• qwen_qwen3-235b-a22b-2507: 0.563
• anthropic_claude-haiku-4.5: 0.563
• z-ai_glm-4.7: 0.563
• openai_gpt-5-mini_reasoning_high: 0.549
• meta-llama_llama-3-70b-instruct: 0.535
• kwaipilot_kat-coder-pro_free: 0.535
• openai_gpt-5-mini_reasoning_low: 0.535
• anthropic_claude-3.5-haiku: 0.507
• x-ai_grok-4.1-fast_reasoning_none: 0.507
• openai_gpt-4o-mini: 0.493
• qwen_qwen3-32b: 0.423
• google_gemma-2-9b-it: 0.352
• qwen_qwen3-8b: 0.352
• meta-llama_llama-3-8b-instruct: 0.296
• mistralai_ministral-8b: 0.282
• mistralai_mistral-7b-instruct-v0.1: 0.211

Overview

Tests formal logic knowledge and principles. Queries cover logical fallacies (ad novitatem, disjunctive syllogism, complex question fallacy), deductive reasoning principles (valid argument structures, relationship between premises and conclusions), and advanced mathematical logic (Kripke countermodels for intuitionistic propositional logic). Evaluates understanding of formal logic terminology, the ability to identify fallacious reasoning, and knowledge of both classical and non-classical logic systems. Distinct from the broader 'reasoning' capability by focusing specifically on formal logical structures and principles.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-flash-preview_reasoning_high: 0.913
• openai_gpt-5.2_reasoning_high: 0.891
• anthropic_claude-sonnet-4.5_reasoning_high: 0.891
• deepseek_deepseek-v3.2-speciale: 0.870
• openai_gpt-5-mini_reasoning_high: 0.870
• openai_gpt-5.2_reasoning_low: 0.870
• x-ai_grok-4.1-fast_reasoning_low: 0.870
• anthropic_claude-sonnet-4.5_reasoning_low: 0.870
• x-ai_grok-4.1-fast_reasoning_high: 0.870
• google_gemini-3-pro-preview_reasoning_low: 0.870
• google_gemini-3-pro-preview_reasoning_high: 0.870
• google_gemini-3-flash-preview_reasoning_low: 0.848
• openai_gpt-5-mini_reasoning_low: 0.826
• openai_gpt-5-nano_reasoning_high: 0.804
• anthropic_claude-sonnet-4.5_reasoning_none: 0.804
• openai_gpt-5-nano_reasoning_low: 0.783
• qwen_qwen3-32b: 0.761
• openai_gpt-5-nano: 0.761
• qwen_qwen3-8b: 0.739
• qwen_qwen3-235b-a22b-2507: 0.739
• x-ai_grok-4.1-fast_reasoning_none: 0.717
• anthropic_claude-haiku-4.5: 0.696
• z-ai_glm-4.7: 0.696
• openai_gpt-4o: 0.543
• kwaipilot_kat-coder-pro_free: 0.543
• deepseek_deepseek-v3.2-exp: 0.478
• meta-llama_llama-3.3-70b-instruct: 0.457
• google_gemini-2.0-flash-001: 0.457
• mistralai_devstral-2512_free: 0.457
• anthropic_claude-3.5-haiku: 0.413
• meta-llama_llama-3-8b-instruct: 0.391
• meta-llama_llama-3-70b-instruct: 0.391
• mistralai_mistral-7b-instruct-v0.1: 0.348
• openai_gpt-4o-mini: 0.348
• google_gemma-2-9b-it: 0.326
• mistralai_ministral-8b: 0.326

Overview

Tests whether models can retrieve and reason over information buried within very long contexts (64K-128K tokens). Unlike simple needle-in-haystack tests that use literal string matching, this capability requires models to infer semantic connections between questions and distant context. Evaluates if models maintain retrieval accuracy as context length increases and relevant information becomes harder to locate through attention mechanisms alone.

Evaluation Method

Embeds a factual statement (the 'needle') at a specific position within 64K-128K tokens of book text (the 'haystack'). The needle contains information needed to answer a question, but the question and needle are written with different vocabulary and phrasing to prevent simple keyword matching (e.g., question asks 'What year did the protagonist visit Paris?' while needle states 'In 1889, Jean traveled to the French capital'). Needle placement varies across queries to test retrieval at different context depths. Models must locate the needle through semantic understanding rather than lexical overlap, then extract the answer from it.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer (after normalization), otherwise 0.0. Uses exact match grading with case-insensitive comparison. Performance typically degrades as context length increases due to attention mechanism challenges in retrieving information without literal string matches between question and needle.

Model Scores

• openai_gpt-5-mini_reasoning_high: 0.453
• google_gemini-3-pro-preview_reasoning_high: 0.448
• openai_gpt-5.2_reasoning_high: 0.446
• google_gemini-3-flash-preview_reasoning_high: 0.397
• google_gemini-3-pro-preview_reasoning_low: 0.387
• openai_gpt-5.2_reasoning_low: 0.381
• openai_gpt-5-mini_reasoning_low: 0.362
• x-ai_grok-4.1-fast_reasoning_high: 0.341
• openai_gpt-5-nano_reasoning_high: 0.327
• google_gemini-3-flash-preview_reasoning_low: 0.316
• x-ai_grok-4.1-fast_reasoning_low: 0.313
• anthropic_claude-sonnet-4.5_reasoning_high: 0.280
• openai_gpt-5-nano: 0.260
• kwaipilot_kat-coder-pro_free: 0.257
• anthropic_claude-sonnet-4.5_reasoning_low: 0.244
• openai_gpt-5-nano_reasoning_low: 0.237
• deepseek_deepseek-v3.2-exp: 0.210
• deepseek_deepseek-v3.2-speciale: 0.201
• anthropic_claude-sonnet-4.5_reasoning_none: 0.201
• x-ai_grok-4.1-fast_reasoning_none: 0.185
• qwen_qwen3-235b-a22b-2507: 0.175
• mistralai_devstral-2512_free: 0.147
• openai_gpt-4o: 0.143
• anthropic_claude-haiku-4.5: 0.136
• qwen_qwen3-8b: 0.132
• google_gemini-2.0-flash-001: 0.132
• z-ai_glm-4.7: 0.128
• anthropic_claude-3.5-haiku: 0.121
• qwen_qwen3-32b: 0.076
• openai_gpt-4o-mini: 0.046
• meta-llama_llama-3.3-70b-instruct: 0.045
• mistralai_ministral-8b: 0.038
• mistralai_mistral-7b-instruct-v0.1: 0.000
• google_gemma-2-9b-it: 0.000
• meta-llama_llama-3-8b-instruct: 0.000
• meta-llama_llama-3-70b-instruct: 0.000

Overview

Tests mathematics problems and knowledge across multiple levels and domains. Covers foundational topics (algebraic manipulation, equations, inequalities, fundamental mathematical concepts) to advanced university-level mathematics including abstract algebra, group theory, ring theory, and field theory. Queries span computational problem-solving, mathematical reasoning, theoretical understanding of mathematical structures, and problem-solving across various mathematical domains including algebra, calculus, number theory, and other core mathematical areas.

Evaluation Method

Uses both numeric matching for computational problems and exact match for multiple choice theory questions. Computational problems require extracting and matching numeric values, while theory questions use multiple choice format with exact string matching.

Scoring

For numeric questions: returns score 1.0 if the extracted numeric value matches the expected value within the specified tolerance (evaluation_criteria.tolerance), otherwise 0.0. Tolerance can be absolute (e.g., 0.01 for +/- 0.01) or relative (e.g., 0.01 for 1% when evaluation_criteria.relative=True). Defaults to 1e-9 if no tolerance is specified. Handles commas, dollar signs, and other formatting in extraction. For multiple choice questions: returns score 1.0 if the extracted answer letter exactly matches the expected answer (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices. The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.826
• openai_gpt-5-mini_reasoning_high: 0.804
• x-ai_grok-4.1-fast_reasoning_high: 0.804
• x-ai_grok-4.1-fast_reasoning_low: 0.783
• openai_gpt-5.2_reasoning_high: 0.783
• google_gemini-3-flash-preview_reasoning_high: 0.783
• deepseek_deepseek-v3.2-speciale: 0.761
• openai_gpt-5-nano_reasoning_high: 0.761
• openai_gpt-5.2_reasoning_low: 0.761
• anthropic_claude-sonnet-4.5_reasoning_low: 0.761
• anthropic_claude-sonnet-4.5_reasoning_high: 0.761
• openai_gpt-5-mini_reasoning_low: 0.739
• openai_gpt-5-nano_reasoning_low: 0.739
• openai_gpt-5-nano: 0.717
• google_gemini-3-flash-preview_reasoning_low: 0.717
• z-ai_glm-4.7: 0.652
• qwen_qwen3-235b-a22b-2507: 0.630
• qwen_qwen3-32b: 0.609
• anthropic_claude-sonnet-4.5_reasoning_none: 0.565
• google_gemini-3-pro-preview_reasoning_low: 0.565
• qwen_qwen3-8b: 0.543
• kwaipilot_kat-coder-pro_free: 0.435
• anthropic_claude-haiku-4.5: 0.283
• google_gemini-2.0-flash-001: 0.239
• x-ai_grok-4.1-fast_reasoning_none: 0.239
• deepseek_deepseek-v3.2-exp: 0.196
• mistralai_devstral-2512_free: 0.130
• meta-llama_llama-3-8b-instruct: 0.087
• meta-llama_llama-3-70b-instruct: 0.087
• meta-llama_llama-3.3-70b-instruct: 0.087
• anthropic_claude-3.5-haiku: 0.087
• openai_gpt-4o: 0.087
• mistralai_mistral-7b-instruct-v0.1: 0.065
• google_gemma-2-9b-it: 0.065
• openai_gpt-4o-mini: 0.065
• mistralai_ministral-8b: 0.022

Overview

Tests medical knowledge and understanding, including anatomy, physiology, medical conditions, treatments, and medical reasoning. Queries evaluate understanding of medical concepts and applications.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• openai_gpt-5.2_reasoning_high: 0.806
• anthropic_claude-sonnet-4.5_reasoning_high: 0.778
• openai_gpt-5.2_reasoning_low: 0.750
• x-ai_grok-4.1-fast_reasoning_low: 0.750
• x-ai_grok-4.1-fast_reasoning_high: 0.750
• google_gemini-3-flash-preview_reasoning_high: 0.750
• deepseek_deepseek-v3.2-speciale: 0.722
• openai_gpt-5-nano_reasoning_high: 0.722
• google_gemini-3-flash-preview_reasoning_low: 0.722
• openai_gpt-5-nano: 0.694
• openai_gpt-5-nano_reasoning_low: 0.694
• openai_gpt-5-mini_reasoning_high: 0.694
• anthropic_claude-sonnet-4.5_reasoning_low: 0.694
• anthropic_claude-sonnet-4.5_reasoning_none: 0.667
• z-ai_glm-4.7: 0.639
• openai_gpt-5-mini_reasoning_low: 0.639
• google_gemini-3-pro-preview_reasoning_low: 0.639
• google_gemini-3-pro-preview_reasoning_high: 0.639
• qwen_qwen3-32b: 0.611
• qwen_qwen3-235b-a22b-2507: 0.611
• anthropic_claude-haiku-4.5: 0.611
• x-ai_grok-4.1-fast_reasoning_none: 0.556
• qwen_qwen3-8b: 0.528
• kwaipilot_kat-coder-pro_free: 0.528
• google_gemini-2.0-flash-001: 0.472
• deepseek_deepseek-v3.2-exp: 0.472
• openai_gpt-4o-mini: 0.444
• meta-llama_llama-3.3-70b-instruct: 0.417
• anthropic_claude-3.5-haiku: 0.417
• openai_gpt-4o: 0.417
• mistralai_devstral-2512_free: 0.417
• meta-llama_llama-3-70b-instruct: 0.389
• mistralai_mistral-7b-instruct-v0.1: 0.306
• meta-llama_llama-3-8b-instruct: 0.250
• google_gemma-2-9b-it: 0.222
• mistralai_ministral-8b: 0.194

Overview

Tests neuroscience knowledge and understanding, including brain structure, neural processes, cognitive neuroscience, and neurological systems. Queries evaluate understanding of how the nervous system works.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• anthropic_claude-sonnet-4.5_reasoning_high: 0.733
• google_gemini-3-pro-preview_reasoning_high: 0.733
• openai_gpt-5.2_reasoning_high: 0.700
• x-ai_grok-4.1-fast_reasoning_high: 0.667
• openai_gpt-5.2_reasoning_low: 0.633
• x-ai_grok-4.1-fast_reasoning_low: 0.633
• anthropic_claude-sonnet-4.5_reasoning_none: 0.600
• openai_gpt-5-mini_reasoning_high: 0.600
• deepseek_deepseek-v3.2-speciale: 0.567
• google_gemini-3-flash-preview_reasoning_low: 0.567
• anthropic_claude-sonnet-4.5_reasoning_low: 0.567
• google_gemini-3-flash-preview_reasoning_high: 0.567
• openai_gpt-5-mini_reasoning_low: 0.533
• google_gemini-3-pro-preview_reasoning_low: 0.533
• openai_gpt-4o: 0.500
• qwen_qwen3-235b-a22b-2507: 0.500
• x-ai_grok-4.1-fast_reasoning_none: 0.500
• anthropic_claude-haiku-4.5: 0.500
• openai_gpt-5-nano_reasoning_high: 0.500
• deepseek_deepseek-v3.2-exp: 0.467
• qwen_qwen3-8b: 0.433
• qwen_qwen3-32b: 0.433
• openai_gpt-5-nano: 0.433
• openai_gpt-5-nano_reasoning_low: 0.433
• mistralai_devstral-2512_free: 0.400
• kwaipilot_kat-coder-pro_free: 0.400
• meta-llama_llama-3.3-70b-instruct: 0.367
• anthropic_claude-3.5-haiku: 0.367
• mistralai_ministral-8b: 0.333
• meta-llama_llama-3-70b-instruct: 0.333
• openai_gpt-4o-mini: 0.300
• google_gemini-2.0-flash-001: 0.300
• z-ai_glm-4.7: 0.300
• mistralai_mistral-7b-instruct-v0.1: 0.267
• meta-llama_llama-3-8b-instruct: 0.267
• google_gemma-2-9b-it: 0.200

Overview

Tests nutrition knowledge and understanding, including nutritional science, dietary principles, food science, and nutritional applications. Queries evaluate understanding of nutrition concepts.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• anthropic_claude-sonnet-4.5_reasoning_high: 0.806
• google_gemini-3-pro-preview_reasoning_high: 0.806
• anthropic_claude-sonnet-4.5_reasoning_low: 0.742
• google_gemini-3-flash-preview_reasoning_low: 0.710
• google_gemini-3-flash-preview_reasoning_high: 0.710
• x-ai_grok-4.1-fast_reasoning_high: 0.677
• anthropic_claude-sonnet-4.5_reasoning_none: 0.645
• google_gemini-3-pro-preview_reasoning_low: 0.645
• openai_gpt-5.2_reasoning_high: 0.613
• openai_gpt-5.2_reasoning_low: 0.581
• x-ai_grok-4.1-fast_reasoning_low: 0.581
• z-ai_glm-4.7: 0.548
• kwaipilot_kat-coder-pro_free: 0.516
• google_gemini-2.0-flash-001: 0.452
• deepseek_deepseek-v3.2-speciale: 0.452
• anthropic_claude-haiku-4.5: 0.419
• openai_gpt-5-mini_reasoning_high: 0.419
• mistralai_ministral-8b: 0.387
• qwen_qwen3-32b: 0.387
• mistralai_devstral-2512_free: 0.387
• qwen_qwen3-235b-a22b-2507: 0.387
• x-ai_grok-4.1-fast_reasoning_none: 0.355
• openai_gpt-5-nano_reasoning_high: 0.355
• anthropic_claude-3.5-haiku: 0.323
• openai_gpt-5-nano: 0.323
• openai_gpt-5-mini_reasoning_low: 0.323
• openai_gpt-5-nano_reasoning_low: 0.323
• mistralai_mistral-7b-instruct-v0.1: 0.290
• meta-llama_llama-3-8b-instruct: 0.290
• meta-llama_llama-3-70b-instruct: 0.290
• qwen_qwen3-8b: 0.290
• meta-llama_llama-3.3-70b-instruct: 0.226
• deepseek_deepseek-v3.2-exp: 0.161
• openai_gpt-4o-mini: 0.129
• openai_gpt-4o: 0.129
• google_gemma-2-9b-it: 0.097

Overview

Tests philosophy knowledge and understanding, including philosophical reasoning, ethical theories, philosophical arguments, and philosophical concepts. Queries evaluate philosophical thinking and analysis.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.900
• google_gemini-3-pro-preview_reasoning_low: 0.833
• google_gemini-3-flash-preview_reasoning_low: 0.767
• google_gemini-3-flash-preview_reasoning_high: 0.733
• anthropic_claude-sonnet-4.5_reasoning_high: 0.733
• anthropic_claude-sonnet-4.5_reasoning_low: 0.700
• anthropic_claude-sonnet-4.5_reasoning_none: 0.633
• openai_gpt-5.2_reasoning_high: 0.633
• openai_gpt-5.2_reasoning_low: 0.600
• x-ai_grok-4.1-fast_reasoning_high: 0.533
• openai_gpt-5-mini_reasoning_high: 0.500
• x-ai_grok-4.1-fast_reasoning_low: 0.467
• openai_gpt-5-nano_reasoning_high: 0.433
• z-ai_glm-4.7: 0.400
• openai_gpt-5-nano_reasoning_low: 0.400
• openai_gpt-4o: 0.367
• google_gemini-2.0-flash-001: 0.367
• x-ai_grok-4.1-fast_reasoning_none: 0.367
• anthropic_claude-haiku-4.5: 0.367
• kwaipilot_kat-coder-pro_free: 0.367
• openai_gpt-5-nano: 0.367
• deepseek_deepseek-v3.2-speciale: 0.367
• openai_gpt-5-mini_reasoning_low: 0.333
• mistralai_devstral-2512_free: 0.267
• mistralai_mistral-7b-instruct-v0.1: 0.233
• meta-llama_llama-3.3-70b-instruct: 0.233
• qwen_qwen3-8b: 0.233
• qwen_qwen3-32b: 0.233
• qwen_qwen3-235b-a22b-2507: 0.200
• openai_gpt-4o-mini: 0.167
• meta-llama_llama-3-8b-instruct: 0.133
• mistralai_ministral-8b: 0.133
• meta-llama_llama-3-70b-instruct: 0.133
• deepseek_deepseek-v3.2-exp: 0.133
• anthropic_claude-3.5-haiku: 0.100
• google_gemma-2-9b-it: 0.033

Overview

Tests physics knowledge and understanding, including mechanics, thermodynamics, electromagnetism, quantum physics, and physical principles. Queries evaluate understanding of physical laws and their applications.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• anthropic_claude-sonnet-4.5_reasoning_high: 0.652
• openai_gpt-5.2_reasoning_low: 0.630
• openai_gpt-5.2_reasoning_high: 0.630
• google_gemini-3-flash-preview_reasoning_low: 0.609
• google_gemini-3-flash-preview_reasoning_high: 0.609
• openai_gpt-5-mini_reasoning_low: 0.587
• openai_gpt-5-mini_reasoning_high: 0.587
• google_gemini-3-pro-preview_reasoning_low: 0.587
• google_gemini-3-pro-preview_reasoning_high: 0.565
• deepseek_deepseek-v3.2-speciale: 0.543
• anthropic_claude-sonnet-4.5_reasoning_none: 0.543
• anthropic_claude-sonnet-4.5_reasoning_low: 0.543
• openai_gpt-5-nano: 0.522
• openai_gpt-5-nano_reasoning_low: 0.522
• openai_gpt-5-nano_reasoning_high: 0.522
• x-ai_grok-4.1-fast_reasoning_low: 0.500
• x-ai_grok-4.1-fast_reasoning_high: 0.500
• qwen_qwen3-235b-a22b-2507: 0.478
• kwaipilot_kat-coder-pro_free: 0.478
• google_gemini-2.0-flash-001: 0.457
• mistralai_devstral-2512_free: 0.457
• z-ai_glm-4.7: 0.457
• deepseek_deepseek-v3.2-exp: 0.435
• qwen_qwen3-8b: 0.413
• qwen_qwen3-32b: 0.413
• openai_gpt-4o: 0.413
• meta-llama_llama-3.3-70b-instruct: 0.348
• anthropic_claude-haiku-4.5: 0.348
• meta-llama_llama-3-8b-instruct: 0.283
• mistralai_ministral-8b: 0.283
• meta-llama_llama-3-70b-instruct: 0.283
• x-ai_grok-4.1-fast_reasoning_none: 0.283
• openai_gpt-4o-mini: 0.239
• google_gemma-2-9b-it: 0.217
• anthropic_claude-3.5-haiku: 0.196
• mistralai_mistral-7b-instruct-v0.1: 0.152

Overview

Tests psychology knowledge and understanding, including cognitive psychology, behavioral psychology, psychological theories, and psychological processes. Queries evaluate understanding of human psychology.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.714
• anthropic_claude-sonnet-4.5_reasoning_high: 0.657
• anthropic_claude-sonnet-4.5_reasoning_low: 0.629
• x-ai_grok-4.1-fast_reasoning_high: 0.629
• google_gemini-3-pro-preview_reasoning_low: 0.629
• anthropic_claude-sonnet-4.5_reasoning_none: 0.600
• x-ai_grok-4.1-fast_reasoning_low: 0.600
• openai_gpt-5.2_reasoning_high: 0.600
• openai_gpt-5-mini_reasoning_high: 0.571
• openai_gpt-5.2_reasoning_low: 0.571
• google_gemini-3-flash-preview_reasoning_low: 0.571
• google_gemini-3-flash-preview_reasoning_high: 0.571
• openai_gpt-5-nano_reasoning_high: 0.543
• openai_gpt-5-mini_reasoning_low: 0.514
• kwaipilot_kat-coder-pro_free: 0.486
• openai_gpt-5-nano: 0.486
• openai_gpt-5-nano_reasoning_low: 0.486
• deepseek_deepseek-v3.2-speciale: 0.457
• mistralai_devstral-2512_free: 0.429
• anthropic_claude-haiku-4.5: 0.429
• google_gemini-2.0-flash-001: 0.400
• x-ai_grok-4.1-fast_reasoning_none: 0.400
• openai_gpt-4o: 0.371
• deepseek_deepseek-v3.2-exp: 0.371
• qwen_qwen3-235b-a22b-2507: 0.343
• mistralai_mistral-7b-instruct-v0.1: 0.314
• meta-llama_llama-3-8b-instruct: 0.314
• meta-llama_llama-3-70b-instruct: 0.314
• meta-llama_llama-3.3-70b-instruct: 0.314
• z-ai_glm-4.7: 0.314
• google_gemma-2-9b-it: 0.286
• mistralai_ministral-8b: 0.286
• openai_gpt-4o-mini: 0.286
• anthropic_claude-3.5-haiku: 0.286
• qwen_qwen3-8b: 0.286
• qwen_qwen3-32b: 0.286

Overview

Tests public relations knowledge and understanding, including communication strategies, crisis management, media relations, and PR principles. Queries evaluate understanding of public relations practices.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-flash-preview_reasoning_low: 0.630
• google_gemini-3-flash-preview_reasoning_high: 0.630
• google_gemini-3-pro-preview_reasoning_high: 0.630
• anthropic_claude-sonnet-4.5_reasoning_none: 0.593
• anthropic_claude-sonnet-4.5_reasoning_low: 0.593
• openai_gpt-5.2_reasoning_high: 0.593
• anthropic_claude-sonnet-4.5_reasoning_high: 0.593
• mistralai_devstral-2512_free: 0.556
• google_gemini-3-pro-preview_reasoning_low: 0.556
• z-ai_glm-4.7: 0.519
• openai_gpt-5.2_reasoning_low: 0.519
• openai_gpt-4o: 0.481
• google_gemini-2.0-flash-001: 0.481
• kwaipilot_kat-coder-pro_free: 0.481
• openai_gpt-5-mini_reasoning_high: 0.481
• x-ai_grok-4.1-fast_reasoning_high: 0.481
• anthropic_claude-haiku-4.5: 0.444
• mistralai_ministral-8b: 0.407
• openai_gpt-5-mini_reasoning_low: 0.407
• openai_gpt-5-nano_reasoning_high: 0.407
• mistralai_mistral-7b-instruct-v0.1: 0.370
• qwen_qwen3-32b: 0.370
• qwen_qwen3-235b-a22b-2507: 0.370
• x-ai_grok-4.1-fast_reasoning_none: 0.370
• openai_gpt-5-nano: 0.370
• openai_gpt-5-nano_reasoning_low: 0.370
• x-ai_grok-4.1-fast_reasoning_low: 0.370
• openai_gpt-4o-mini: 0.333
• qwen_qwen3-8b: 0.333
• deepseek_deepseek-v3.2-exp: 0.333
• meta-llama_llama-3.3-70b-instruct: 0.296
• anthropic_claude-3.5-haiku: 0.296
• google_gemma-2-9b-it: 0.259
• deepseek_deepseek-v3.2-speciale: 0.259
• meta-llama_llama-3-8b-instruct: 0.222
• meta-llama_llama-3-70b-instruct: 0.222

Overview

Tests puzzle-solving and logical reasoning. Queries present various types of puzzles requiring logical thinking, pattern recognition, and problem-solving skills.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.935
• openai_gpt-5-mini_reasoning_high: 0.903
• x-ai_grok-4.1-fast_reasoning_high: 0.903
• openai_gpt-5.2_reasoning_high: 0.903
• openai_gpt-5.2_reasoning_low: 0.871
• x-ai_grok-4.1-fast_reasoning_low: 0.871
• google_gemini-3-pro-preview_reasoning_low: 0.871
• google_gemini-3-flash-preview_reasoning_high: 0.839
• anthropic_claude-sonnet-4.5_reasoning_high: 0.839
• deepseek_deepseek-v3.2-speciale: 0.806
• openai_gpt-5-mini_reasoning_low: 0.806
• google_gemini-3-flash-preview_reasoning_low: 0.806
• anthropic_claude-sonnet-4.5_reasoning_low: 0.806
• openai_gpt-5-nano: 0.774
• openai_gpt-5-nano_reasoning_low: 0.774
• openai_gpt-5-nano_reasoning_high: 0.774
• anthropic_claude-sonnet-4.5_reasoning_none: 0.774
• z-ai_glm-4.7: 0.613
• qwen_qwen3-32b: 0.581
• mistralai_devstral-2512_free: 0.516
• x-ai_grok-4.1-fast_reasoning_none: 0.516
• anthropic_claude-haiku-4.5: 0.452
• qwen_qwen3-8b: 0.419
• qwen_qwen3-235b-a22b-2507: 0.419
• openai_gpt-4o: 0.355
• google_gemini-2.0-flash-001: 0.355
• deepseek_deepseek-v3.2-exp: 0.290
• meta-llama_llama-3-70b-instruct: 0.226
• kwaipilot_kat-coder-pro_free: 0.226
• anthropic_claude-3.5-haiku: 0.194
• google_gemma-2-9b-it: 0.129
• meta-llama_llama-3-8b-instruct: 0.097
• meta-llama_llama-3.3-70b-instruct: 0.097
• mistralai_mistral-7b-instruct-v0.1: 0.065
• mistralai_ministral-8b: 0.065
• openai_gpt-4o-mini: 0.065

Overview

Tests diverse reasoning capabilities across multiple domains. Queries include commonsense reasoning (e.g., sarcasm detection in social media posts), moral reasoning (ethical philosophy and decision-making), linguistic reasoning (pronoun disambiguation, adjective ordering rules in variant languages), and complex logical reasoning (constraint satisfaction puzzles with 100+ clues, rule-based inference with preference ordering, board game logic, boolean expression evaluation). Evaluates the model's ability to apply appropriate reasoning strategies across contexts, from social understanding to formal logic to complex multi-constraint problem-solving.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• anthropic_claude-sonnet-4.5_reasoning_none: 0.478
• google_gemini-3-flash-preview_reasoning_high: 0.457
• anthropic_claude-sonnet-4.5_reasoning_high: 0.457
• kwaipilot_kat-coder-pro_free: 0.435
• anthropic_claude-sonnet-4.5_reasoning_low: 0.413
• openai_gpt-5-mini_reasoning_high: 0.391
• google_gemini-3-flash-preview_reasoning_low: 0.370
• google_gemini-3-pro-preview_reasoning_low: 0.370
• google_gemini-2.0-flash-001: 0.348
• openai_gpt-5.2_reasoning_low: 0.326
• x-ai_grok-4.1-fast_reasoning_low: 0.326
• openai_gpt-5.2_reasoning_high: 0.326
• google_gemini-3-pro-preview_reasoning_high: 0.326
• openai_gpt-4o: 0.304
• mistralai_devstral-2512_free: 0.304
• qwen_qwen3-235b-a22b-2507: 0.304
• openai_gpt-5-mini_reasoning_low: 0.304
• openai_gpt-5-nano_reasoning_high: 0.304
• x-ai_grok-4.1-fast_reasoning_high: 0.304
• meta-llama_llama-3-70b-instruct: 0.283
• anthropic_claude-3.5-haiku: 0.283
• deepseek_deepseek-v3.2-exp: 0.283
• anthropic_claude-haiku-4.5: 0.283
• openai_gpt-5-nano: 0.283
• openai_gpt-5-nano_reasoning_low: 0.283
• x-ai_grok-4.1-fast_reasoning_none: 0.261
• z-ai_glm-4.7: 0.261
• deepseek_deepseek-v3.2-speciale: 0.261
• mistralai_mistral-7b-instruct-v0.1: 0.239
• mistralai_ministral-8b: 0.239
• qwen_qwen3-8b: 0.239
• google_gemma-2-9b-it: 0.196
• qwen_qwen3-32b: 0.196
• openai_gpt-4o-mini: 0.174
• meta-llama_llama-3.3-70b-instruct: 0.174
• meta-llama_llama-3-8b-instruct: 0.130

Overview

Tests security studies knowledge and understanding, including cybersecurity, information security, security policies, and security practices. Queries evaluate understanding of security concepts.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.774
• google_gemini-3-pro-preview_reasoning_low: 0.742
• anthropic_claude-sonnet-4.5_reasoning_low: 0.710
• anthropic_claude-sonnet-4.5_reasoning_high: 0.710
• x-ai_grok-4.1-fast_reasoning_low: 0.677
• x-ai_grok-4.1-fast_reasoning_high: 0.677
• openai_gpt-5.2_reasoning_high: 0.613
• google_gemini-3-flash-preview_reasoning_high: 0.613
• deepseek_deepseek-v3.2-speciale: 0.581
• anthropic_claude-sonnet-4.5_reasoning_none: 0.581
• openai_gpt-5-mini_reasoning_high: 0.581
• openai_gpt-5.2_reasoning_low: 0.581
• google_gemini-3-flash-preview_reasoning_low: 0.581
• kwaipilot_kat-coder-pro_free: 0.484
• openai_gpt-5-mini_reasoning_low: 0.484
• mistralai_mistral-7b-instruct-v0.1: 0.452
• mistralai_devstral-2512_free: 0.452
• qwen_qwen3-235b-a22b-2507: 0.452
• x-ai_grok-4.1-fast_reasoning_none: 0.452
• anthropic_claude-haiku-4.5: 0.419
• openai_gpt-5-nano: 0.419
• openai_gpt-5-nano_reasoning_low: 0.419
• openai_gpt-5-nano_reasoning_high: 0.419
• anthropic_claude-3.5-haiku: 0.387
• qwen_qwen3-8b: 0.387
• deepseek_deepseek-v3.2-exp: 0.387
• z-ai_glm-4.7: 0.387
• qwen_qwen3-32b: 0.355
• google_gemma-2-9b-it: 0.323
• mistralai_ministral-8b: 0.323
• meta-llama_llama-3-70b-instruct: 0.290
• meta-llama_llama-3.3-70b-instruct: 0.226
• openai_gpt-4o: 0.226
• google_gemini-2.0-flash-001: 0.194
• meta-llama_llama-3-8b-instruct: 0.161
• openai_gpt-4o-mini: 0.129

Overview

Tests social calibration and alignment with user preferences. Queries evaluate the model's ability to understand social contexts, boundaries, appropriate behavior, and adapt responses to user preferences while maintaining appropriate standards.

Evaluation Method

Grades social calibration responses using LLM judges with structured tool-based output. To mitigate single-model bias, each response is evaluated by two independent judge models from different providers (gpt-5-mini and grok-4.1-fast), and scores are averaged. Criteria are designed to be explicit and observable (e.g., 'acknowledges the user's emotional state', 'avoids unsolicited advice') rather than subjective assessments of social intelligence. Evaluates ability to understand social contexts, boundaries, and appropriate behavior.

Scoring

Each judge scores multiple metrics on 1-10 scale using structured tool output. Positive criteria: higher is better. Negative criteria: scores are inverted (a score of 1 becomes 10, a score of 10 becomes 1) so that lower presence of negative behaviors results in higher scores. Scores from both judges (gpt-5-mini and grok-4.1-fast) are averaged and normalized to 0.0-1.0 range. Multi-judge averaging reduces bias from any single model's interpretation of social norms.

Model Scores

• anthropic_claude-haiku-4.5: 0.922
• anthropic_claude-sonnet-4.5_reasoning_none: 0.896
• anthropic_claude-sonnet-4.5_reasoning_high: 0.890
• anthropic_claude-sonnet-4.5_reasoning_low: 0.879
• anthropic_claude-3.5-haiku: 0.867
• openai_gpt-5.2_reasoning_high: 0.819
• openai_gpt-5.2_reasoning_low: 0.816
• google_gemini-3-pro-preview_reasoning_low: 0.787
• google_gemini-3-pro-preview_reasoning_high: 0.778
• z-ai_glm-4.7: 0.773
• google_gemini-3-flash-preview_reasoning_high: 0.726
• qwen_qwen3-235b-a22b-2507: 0.723
• google_gemini-3-flash-preview_reasoning_low: 0.722
• openai_gpt-5-mini_reasoning_low: 0.702
• deepseek_deepseek-v3.2-exp: 0.694
• openai_gpt-5-mini_reasoning_high: 0.684
• kwaipilot_kat-coder-pro_free: 0.681
• deepseek_deepseek-v3.2-speciale: 0.663
• meta-llama_llama-3-70b-instruct: 0.637
• openai_gpt-5-nano_reasoning_low: 0.637
• x-ai_grok-4.1-fast_reasoning_high: 0.633
• openai_gpt-5-nano_reasoning_high: 0.625
• google_gemini-2.0-flash-001: 0.602
• mistralai_devstral-2512_free: 0.601
• x-ai_grok-4.1-fast_reasoning_low: 0.601
• google_gemma-2-9b-it: 0.598
• openai_gpt-5-nano: 0.589
• meta-llama_llama-3.3-70b-instruct: 0.588
• x-ai_grok-4.1-fast_reasoning_none: 0.531
• openai_gpt-4o: 0.506
• qwen_qwen3-32b: 0.448
• qwen_qwen3-8b: 0.434
• meta-llama_llama-3-8b-instruct: 0.406
• mistralai_mistral-7b-instruct-v0.1: 0.371
• openai_gpt-4o-mini: 0.342
• mistralai_ministral-8b: 0.283

Overview

Tests sociology knowledge and understanding, including social structures, social processes, social theories, and sociological analysis. Queries evaluate understanding of social phenomena.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.833
• google_gemini-3-flash-preview_reasoning_low: 0.733
• x-ai_grok-4.1-fast_reasoning_high: 0.700
• anthropic_claude-sonnet-4.5_reasoning_low: 0.667
• google_gemini-3-pro-preview_reasoning_low: 0.667
• google_gemini-3-flash-preview_reasoning_high: 0.667
• anthropic_claude-sonnet-4.5_reasoning_high: 0.667
• x-ai_grok-4.1-fast_reasoning_low: 0.633
• openai_gpt-5.2_reasoning_high: 0.633
• openai_gpt-4o: 0.567
• z-ai_glm-4.7: 0.567
• deepseek_deepseek-v3.2-speciale: 0.567
• anthropic_claude-sonnet-4.5_reasoning_none: 0.567
• openai_gpt-5.2_reasoning_low: 0.567
• openai_gpt-5-nano: 0.533
• openai_gpt-5-nano_reasoning_low: 0.533
• openai_gpt-5-nano_reasoning_high: 0.533
• openai_gpt-5-mini_reasoning_high: 0.533
• kwaipilot_kat-coder-pro_free: 0.500
• openai_gpt-5-mini_reasoning_low: 0.500
• meta-llama_llama-3-8b-instruct: 0.467
• qwen_qwen3-32b: 0.467
• deepseek_deepseek-v3.2-exp: 0.467
• x-ai_grok-4.1-fast_reasoning_none: 0.467
• qwen_qwen3-235b-a22b-2507: 0.433
• mistralai_mistral-7b-instruct-v0.1: 0.400
• anthropic_claude-haiku-4.5: 0.400
• qwen_qwen3-8b: 0.367
• mistralai_devstral-2512_free: 0.367
• mistralai_ministral-8b: 0.333
• openai_gpt-4o-mini: 0.333
• google_gemini-2.0-flash-001: 0.333
• google_gemma-2-9b-it: 0.267
• meta-llama_llama-3.3-70b-instruct: 0.267
• meta-llama_llama-3-70b-instruct: 0.200
• anthropic_claude-3.5-haiku: 0.200

Overview

Tests statistics knowledge and problems, including statistical analysis, probability, data interpretation, and statistical reasoning. Queries require understanding of statistical concepts and methods.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-flash-preview_reasoning_low: 0.903
• google_gemini-3-flash-preview_reasoning_high: 0.903
• x-ai_grok-4.1-fast_reasoning_low: 0.871
• x-ai_grok-4.1-fast_reasoning_high: 0.871
• google_gemini-3-pro-preview_reasoning_low: 0.871
• anthropic_claude-sonnet-4.5_reasoning_high: 0.871
• google_gemini-3-pro-preview_reasoning_high: 0.871
• deepseek_deepseek-v3.2-speciale: 0.839
• openai_gpt-5.2_reasoning_low: 0.839
• anthropic_claude-sonnet-4.5_reasoning_low: 0.839
• openai_gpt-5.2_reasoning_high: 0.839
• qwen_qwen3-235b-a22b-2507: 0.806
• z-ai_glm-4.7: 0.806
• openai_gpt-5-nano: 0.806
• openai_gpt-5-mini_reasoning_low: 0.806
• openai_gpt-5-nano_reasoning_low: 0.806
• openai_gpt-5-nano_reasoning_high: 0.806
• anthropic_claude-sonnet-4.5_reasoning_none: 0.806
• openai_gpt-5-mini_reasoning_high: 0.806
• qwen_qwen3-32b: 0.774
• qwen_qwen3-8b: 0.742
• x-ai_grok-4.1-fast_reasoning_none: 0.742
• anthropic_claude-haiku-4.5: 0.677
• kwaipilot_kat-coder-pro_free: 0.677
• google_gemini-2.0-flash-001: 0.613
• deepseek_deepseek-v3.2-exp: 0.581
• mistralai_devstral-2512_free: 0.484
• openai_gpt-4o: 0.419
• openai_gpt-4o-mini: 0.290
• meta-llama_llama-3.3-70b-instruct: 0.290
• anthropic_claude-3.5-haiku: 0.258
• google_gemma-2-9b-it: 0.226
• mistralai_mistral-7b-instruct-v0.1: 0.194
• mistralai_ministral-8b: 0.161
• meta-llama_llama-3-70b-instruct: 0.129
• meta-llama_llama-3-8b-instruct: 0.097

Overview

Tests generation of valid structured formats (JSON, YAML, XML, TOML, CSV). Queries require the model to produce correctly formatted structured data that is both syntactically valid and contains required content elements.

Evaluation Method

Evaluates structured data generation by parsing the response into the target format (JSON, YAML, XML, TOML, CSV) and validating required field paths exist in the parsed structure. Uses dot-notation path navigation to verify field presence (e.g., 'person.name' checks that parsed_json['person']['name'] exists). The grader extracts code blocks if present, parses the content using format-specific parsers, then navigates the parsed structure using the paths specified in evaluation_criteria.required_content. Supports nested objects, array indexing (e.g., 'items[0].name'), wildcards for array iteration (e.g., 'users[*].id'), and format-specific validation (XML element paths, CSV column headers). The validation is path-based on parsed structures, not string pattern matching - the model must generate syntactically valid structured data with the correct nested field hierarchy.

Scoring

Binary scoring with strict all-or-nothing semantics. Returns 1.0 only if: (1) response parses successfully as valid structured data in the target format, AND (2) every path in evaluation_criteria.required_content exists when navigated in the parsed structure. Returns 0.0 if parsing fails or any single required path is missing. For JSON/YAML/TOML: navigates parsed dictionary/list structures using dot notation with dict key lookup and list indexing. For XML: uses element tree navigation with support for attributes (@attr) and indexed children (element[0]). For CSV: validates that all required column names exist in the header row. A response with 9 out of 10 required fields still scores 0.0 - partial credit is never awarded.

Model Scores

• openai_gpt-5.2_reasoning_low: 0.854
• openai_gpt-5.2_reasoning_high: 0.829
• openai_gpt-5-mini_reasoning_low: 0.805
• openai_gpt-5-nano_reasoning_high: 0.805
• openai_gpt-5-mini_reasoning_high: 0.805
• openai_gpt-4o: 0.756
• google_gemini-3-flash-preview_reasoning_low: 0.756
• google_gemini-3-flash-preview_reasoning_high: 0.756
• mistralai_devstral-2512_free: 0.732
• openai_gpt-5-nano: 0.707
• openai_gpt-5-nano_reasoning_low: 0.707
• anthropic_claude-sonnet-4.5_reasoning_low: 0.707
• anthropic_claude-sonnet-4.5_reasoning_high: 0.707
• google_gemini-3-pro-preview_reasoning_high: 0.707
• anthropic_claude-sonnet-4.5_reasoning_none: 0.683
• x-ai_grok-4.1-fast_reasoning_low: 0.683
• x-ai_grok-4.1-fast_reasoning_high: 0.683
• deepseek_deepseek-v3.2-exp: 0.659
• google_gemini-3-pro-preview_reasoning_low: 0.659
• anthropic_claude-haiku-4.5: 0.634
• deepseek_deepseek-v3.2-speciale: 0.634
• z-ai_glm-4.7: 0.610
• kwaipilot_kat-coder-pro_free: 0.561
• google_gemini-2.0-flash-001: 0.537
• x-ai_grok-4.1-fast_reasoning_none: 0.537
• anthropic_claude-3.5-haiku: 0.512
• meta-llama_llama-3.3-70b-instruct: 0.463
• qwen_qwen3-32b: 0.390
• qwen_qwen3-235b-a22b-2507: 0.390
• openai_gpt-4o-mini: 0.341
• meta-llama_llama-3-70b-instruct: 0.293
• mistralai_ministral-8b: 0.171
• qwen_qwen3-8b: 0.146
• google_gemma-2-9b-it: 0.098
• meta-llama_llama-3-8b-instruct: 0.073
• mistralai_mistral-7b-instruct-v0.1: 0.000

Overview

Tests the model's ability to maintain well-reasoned positions when users push back with demonstrably poor ideas or incorrect assumptions. Queries present a multi-turn conversation where the model provides an initial recommendation, then the user challenges it with flawed counterarguments. The model fails if it abandons its original sound position without good reason, inappropriately agreeing that the user's problematic idea or incorrect assumptions are valid. Success requires maintaining independent judgment, providing thoughtful counterarguments, and resisting the temptation to agree just to please the user. Evaluated using LLM judge with structured criteria on maintaining original position and avoiding sycophantic agreement.

Evaluation Method

Grades sycophancy resistance responses using LLM judges with structured tool-based output. To mitigate single-model bias, each response is evaluated by two independent judge models from different providers (gpt-5-mini and grok-4.1-fast), and scores are averaged. Criteria are designed to be explicit and observable (e.g., 'maintains original recommendation without reversal', 'provides specific counterarguments to flawed user reasoning', 'does not validate demonstrably incorrect assumptions') rather than subjective assessments. Using judges from different model families (OpenAI and xAI) helps ensure that sycophantic patterns specific to one training approach don't bias the evaluation. Evaluates ability to maintain independent judgment and resist inappropriately agreeing with user pushback.

Scoring

Each judge scores multiple metrics on 1-10 scale using structured tool output. Positive criteria: higher is better. Negative criteria: scores are inverted (a score of 1 becomes 10, a score of 10 becomes 1) so that lower presence of negative behaviors results in higher scores. Scores from both judges (gpt-5-mini and grok-4.1-fast) are averaged and normalized to 0.0-1.0 range. Multi-judge averaging from different model families reduces bias toward any single provider's definition of appropriate assertiveness.

Model Scores

• anthropic_claude-sonnet-4.5_reasoning_none: 0.874
• anthropic_claude-haiku-4.5: 0.805
• openai_gpt-5-mini_reasoning_low: 0.775
• openai_gpt-5-nano_reasoning_high: 0.760
• anthropic_claude-sonnet-4.5_reasoning_low: 0.758
• anthropic_claude-sonnet-4.5_reasoning_high: 0.755
• openai_gpt-5.2_reasoning_high: 0.746
• openai_gpt-5-nano_reasoning_low: 0.724
• x-ai_grok-4.1-fast_reasoning_low: 0.722
• x-ai_grok-4.1-fast_reasoning_high: 0.712
• meta-llama_llama-3.3-70b-instruct: 0.708
• openai_gpt-5-mini_reasoning_high: 0.701
• openai_gpt-5.2_reasoning_low: 0.686
• mistralai_devstral-2512_free: 0.683
• anthropic_claude-3.5-haiku: 0.668
• google_gemini-2.0-flash-001: 0.665
• x-ai_grok-4.1-fast_reasoning_none: 0.657
• openai_gpt-5-nano: 0.652
• kwaipilot_kat-coder-pro_free: 0.648
• deepseek_deepseek-v3.2-exp: 0.637
• qwen_qwen3-235b-a22b-2507: 0.614
• google_gemini-3-flash-preview_reasoning_low: 0.574
• google_gemini-3-pro-preview_reasoning_low: 0.566
• deepseek_deepseek-v3.2-speciale: 0.547
• google_gemini-3-flash-preview_reasoning_high: 0.546
• qwen_qwen3-8b: 0.541
• google_gemini-3-pro-preview_reasoning_high: 0.523
• z-ai_glm-4.7: 0.517
• openai_gpt-4o: 0.489
• openai_gpt-4o-mini: 0.420
• mistralai_ministral-8b: 0.407
• mistralai_mistral-7b-instruct-v0.1: 0.243
• meta-llama_llama-3-70b-instruct: 0.194
• google_gemma-2-9b-it: 0.190
• qwen_qwen3-32b: 0.163
• meta-llama_llama-3-8b-instruct: 0.154

Overview

Tests whether models prioritize custom safety policies from deployers over their default training behaviors. Evaluates if models can adapt their safety boundaries based on organization-specific guidelines (e.g., enterprise contact protocols, regional content restrictions, industry-specific safety rules) rather than applying one-size-fits-all safety training. Critical for real-world deployments where different organizations have different safety requirements.

Evaluation Method

Injects custom safety policies via system_prompt_addition field (e.g., 'Always provide the SafetyLine number 1-800-XXX-XXXX for crisis situations' or 'Never generate instructions for creating weapons'). Users then attempt to bypass these guidelines using manipulation tactics like social engineering, urgency pressure, or embedding requests in long contexts. Evaluation criteria contain verifiable requirements (e.g., 'response must mention the SafetyLine number' or 'response must refuse the weapon request') that are checked programmatically or via LLM judges. Tests if models follow deployer instructions over base training when the two conflict.

Scoring

Each judge scores multiple metrics on 1-10 scale using structured tool output. Positive criteria: higher is better. Negative criteria: scores are inverted (a score of 1 becomes 10, a score of 10 becomes 1) so that lower presence of negative behaviors results in higher scores. Scores from both judges are averaged and normalized to 0.0-1.0 range. Multi-judge averaging from different providers reduces bias from any single model's safety alignment affecting evaluation.

Model Scores

• x-ai_grok-4.1-fast_reasoning_high: 0.955
• anthropic_claude-sonnet-4.5_reasoning_high: 0.953
• google_gemini-3-flash-preview_reasoning_low: 0.944
• openai_gpt-5.2_reasoning_low: 0.940
• openai_gpt-5-mini_reasoning_high: 0.935
• openai_gpt-5.2_reasoning_high: 0.932
• anthropic_claude-sonnet-4.5_reasoning_low: 0.928
• anthropic_claude-sonnet-4.5_reasoning_none: 0.926
• x-ai_grok-4.1-fast_reasoning_low: 0.923
• openai_gpt-5-mini_reasoning_low: 0.921
• google_gemini-3-flash-preview_reasoning_high: 0.917
• google_gemini-3-pro-preview_reasoning_low: 0.906
• google_gemini-3-pro-preview_reasoning_high: 0.891
• x-ai_grok-4.1-fast_reasoning_none: 0.885
• openai_gpt-5-nano_reasoning_high: 0.882
• openai_gpt-5-nano_reasoning_low: 0.852
• openai_gpt-5-nano: 0.835
• z-ai_glm-4.7: 0.829
• anthropic_claude-haiku-4.5: 0.822
• google_gemini-2.0-flash-001: 0.796
• anthropic_claude-3.5-haiku: 0.787
• mistralai_devstral-2512_free: 0.772
• qwen_qwen3-235b-a22b-2507: 0.761
• deepseek_deepseek-v3.2-speciale: 0.716
• deepseek_deepseek-v3.2-exp: 0.654
• openai_gpt-4o: 0.616
• qwen_qwen3-32b: 0.566
• kwaipilot_kat-coder-pro_free: 0.555
• openai_gpt-4o-mini: 0.549
• qwen_qwen3-8b: 0.538
• meta-llama_llama-3.3-70b-instruct: 0.523
• google_gemma-2-9b-it: 0.460
• mistralai_mistral-7b-instruct-v0.1: 0.406
• mistralai_ministral-8b: 0.389
• meta-llama_llama-3-70b-instruct: 0.359
• meta-llama_llama-3-8b-instruct: 0.312

Overview

Tests tool usage capability including simple, multiple, parallel, and language-specific tool use. Queries require the model to correctly identify when tools are needed, select appropriate tools, format tool calls correctly, and use tools effectively to accomplish tasks.

Evaluation Method

Grades tool calls by comparing against expected function names and parameters. Accepts tool calls in the standard OpenAI/OpenRouter API format (structured JSON with function.name and function.arguments fields).

Scoring

Returns score 1.0 if the number of tool calls matches expected (within min_calls/max_calls range) AND each call matches expected function name and all arguments match valid values. Returns 0.0 otherwise. Handles both string and dict formats for arguments (including double-encoded JSON from some providers). When the answer field contains arrays for argument values (e.g., {"accountNumber": ["FF123456789"]}), this indicates multiple acceptable values for matching purposes. The model does not need to return an array, but its argument value must match one of the values in the array. Parameters with default values in the tool schema may be included or omitted by the model without penalty. If the model explicitly includes a parameter set to its default value (e.g., including "includeExpirationInfo": true when the schema default is true), this is treated as semantically equivalent to omitting that parameter and does not result in a scoring penalty. Supports flexible value matching including type coercion, string normalization, keyword matching, and list subset matching.

Model Scores

• openai_gpt-5-nano_reasoning_high: 0.637
• openai_gpt-4o: 0.625
• openai_gpt-5-nano_reasoning_low: 0.613
• kwaipilot_kat-coder-pro_free: 0.588
• openai_gpt-5-nano: 0.588
• google_gemini-3-flash-preview_reasoning_low: 0.588
• anthropic_claude-sonnet-4.5_reasoning_high: 0.575
• openai_gpt-5.2_reasoning_low: 0.550
• google_gemini-3-flash-preview_reasoning_high: 0.550
• openai_gpt-5-mini_reasoning_low: 0.525
• anthropic_claude-sonnet-4.5_reasoning_low: 0.525
• z-ai_glm-4.7: 0.512
• openai_gpt-5.2_reasoning_high: 0.512
• anthropic_claude-sonnet-4.5_reasoning_none: 0.500
• google_gemini-3-pro-preview_reasoning_high: 0.500
• openai_gpt-5-mini_reasoning_high: 0.487
• openai_gpt-4o-mini: 0.475
• anthropic_claude-3.5-haiku: 0.475
• mistralai_devstral-2512_free: 0.475
• anthropic_claude-haiku-4.5: 0.475
• x-ai_grok-4.1-fast_reasoning_high: 0.475
• qwen_qwen3-235b-a22b-2507: 0.463
• google_gemini-3-pro-preview_reasoning_low: 0.463
• x-ai_grok-4.1-fast_reasoning_low: 0.450
• x-ai_grok-4.1-fast_reasoning_none: 0.438
• deepseek_deepseek-v3.2-exp: 0.412
• qwen_qwen3-8b: 0.350
• mistralai_ministral-8b: 0.312
• qwen_qwen3-32b: 0.300
• google_gemini-2.0-flash-001: 0.250
• meta-llama_llama-3.3-70b-instruct: 0.175
• meta-llama_llama-3-70b-instruct: 0.150
• meta-llama_llama-3-8b-instruct: 0.037
• mistralai_mistral-7b-instruct-v0.1: 0.000
• google_gemma-2-9b-it: 0.000
• deepseek_deepseek-v3.2-speciale: 0.000

Overview

Tests general trivia and factual knowledge across various domains. Queries evaluate the model's breadth of factual knowledge and ability to recall specific information.

Evaluation Method

Grades multiple choice responses by exact string matching with normalization.

Scoring

Returns score 1.0 if the extracted answer exactly matches the expected answer letter (after normalization), otherwise 0.0. The system prompt requests answers in `<answer>X</answer>` format where X is a letter from the provided choices (A, B, C, D, etc.). The grader normalizes for models that include the full choice text instead of just the letter, or that violate the answer tag format from the system prompt. Only one answer is correct.

Model Scores

• google_gemini-3-pro-preview_reasoning_high: 0.833
• google_gemini-3-pro-preview_reasoning_low: 0.733
• google_gemini-3-flash-preview_reasoning_low: 0.667
• anthropic_claude-sonnet-4.5_reasoning_low: 0.667
• anthropic_claude-sonnet-4.5_reasoning_high: 0.667
• google_gemini-3-flash-preview_reasoning_high: 0.633
• openai_gpt-5.2_reasoning_high: 0.600
• anthropic_claude-sonnet-4.5_reasoning_none: 0.533
• deepseek_deepseek-v3.2-speciale: 0.500
• openai_gpt-5.2_reasoning_low: 0.500
• x-ai_grok-4.1-fast_reasoning_low: 0.500
• x-ai_grok-4.1-fast_reasoning_high: 0.500
• mistralai_devstral-2512_free: 0.467
• x-ai_grok-4.1-fast_reasoning_none: 0.467
• anthropic_claude-haiku-4.5: 0.467
• openai_gpt-5-nano: 0.467
• openai_gpt-5-nano_reasoning_low: 0.467
• openai_gpt-5-nano_reasoning_high: 0.467
• openai_gpt-4o: 0.433
• deepseek_deepseek-v3.2-exp: 0.433
• z-ai_glm-4.7: 0.433
• openai_gpt-5-mini_reasoning_low: 0.433
• openai_gpt-5-mini_reasoning_high: 0.433
• meta-llama_llama-3-70b-instruct: 0.400
• meta-llama_llama-3.3-70b-instruct: 0.400
• anthropic_claude-3.5-haiku: 0.400
• qwen_qwen3-235b-a22b-2507: 0.400
• openai_gpt-4o-mini: 0.367
• qwen_qwen3-32b: 0.367
• google_gemini-2.0-flash-001: 0.367
• kwaipilot_kat-coder-pro_free: 0.333
• mistralai_ministral-8b: 0.233
• qwen_qwen3-8b: 0.233
• meta-llama_llama-3-8b-instruct: 0.200
• google_gemma-2-9b-it: 0.133
• mistralai_mistral-7b-instruct-v0.1: 0.100