LLM Evaluation

← Back to main page

Run of 2026-02-10 15:01:24

16
Models Tested
13
Total Tasks
47.1%
Overall Success Rate
98/208
Passed Combinations

Details

Score Model task10_multiple_tests task11_relationship_classifier task12_need_reply task13_meeting_action_items task14_graph_money_distribution task1_file_list task2_fix_python_syntax task4_web_fetch task5_dedup_contact task6_config_merger task7_log_parser task8_regex_extraction task9_cpp_footguns
92.3% openrouter/anthropic/claude-opus-4.5 ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
84.6% openrouter/anthropic/claude-haiku-4.5 ✅ ✅ ✅ ✅ ❌ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅
76.9% openrouter/google/gemini-3-pro-preview ✅ ✅ ✅ ❌ ❌ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅
76.9% openrouter/anthropic/claude-sonnet-4.5 ✅ ✅ ❌ ❌ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
76.9% openrouter/x-ai/grok-code-fast-1 ✅ ✅ ✅ ❌ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌
69.2% openrouter/deepseek/deepseek-v3.1-terminus ✅ ✅ ✅ ❌ ❌ ✅ ✅ ❌ ❌ ✅ ✅ ✅ ✅
61.5% openrouter/google/gemini-2.5-pro ✅ ✅ ✅ ❌ ❌ ✅ ✅ ❌ ✅ ❌ ✅ ✅ ❌
53.8% openrouter/google/gemini-2.5-flash-lite-preview-09-2025 ✅ ❌ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ✅ ✅ ✅ ✅
30.8% openrouter/google/gemini-2.5-flash-preview-09-2025 ❌ ✅ ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌ ✅
30.8% openrouter/openai/gpt-oss-20b ❌ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ❌ ❌ ✅ ✅ ❌
23.1% openrouter/openai/gpt-oss-120b ✅ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ✅ ✅ ❌ ❌
23.1% openrouter/x-ai/grok-3-mini ✅ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ❌ ❌ ❌ ❌ ❌
23.1% openrouter/openai/gpt-4o-mini ✅ ❌ ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌ ❌
15.4% openrouter/qwen/qwen3-coder ✅ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌
15.4% litellm/GLM-4.5-Air-FP8-dev ✅ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌
0.0% openrouter/deepseek/deepseek-chat-v3-0324 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌