LLM Evaluation

← Back to main page

Run of 2026-01-15 15:00:48

15
Models Tested
13
Total Tasks
48.2%
Overall Success Rate
94/195
Passed Combinations

Details

Score Model task10_multiple_tests task11_relationship_classifier task12_need_reply task13_meeting_action_items task14_graph_money_distribution task1_file_list task2_fix_python_syntax task4_web_fetch task5_dedup_contact task6_config_merger task7_log_parser task8_regex_extraction task9_cpp_footguns
84.6% openrouter/anthropic/claude-opus-4.5 ✅ ✅ ❌ ✅ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
84.6% openrouter/anthropic/claude-sonnet-4.5 ✅ ✅ ❌ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅ ✅
76.9% openrouter/anthropic/claude-haiku-4.5 ✅ ✅ ❌ ❌ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅
69.2% openrouter/deepseek/deepseek-v3.1-terminus ✅ ✅ ✅ ❌ ❌ ✅ ❌ ❌ ✅ ✅ ✅ ✅ ✅
69.2% openrouter/x-ai/grok-code-fast-1 ✅ ❌ ✅ ❌ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌
61.5% openrouter/google/gemini-2.5-pro ✅ ✅ ✅ ❌ ❌ ✅ ✅ ❌ ✅ ✅ ❌ ✅ ❌
46.2% openrouter/google/gemini-2.5-flash-lite-preview-09-2025 ✅ ❌ ✅ ❌ ❌ ✅ ✅ ❌ ❌ ✅ ❌ ✅ ❌
38.5% openrouter/x-ai/grok-3-mini ✅ ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ✅ ✅ ✅ ❌
38.5% openrouter/openai/gpt-4o-mini ✅ ❌ ❌ ❌ ❌ ✅ ✅ ❌ ❌ ✅ ❌ ✅ ❌
30.8% openrouter/google/gemini-3-pro-preview ❌ ✅ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌ ❌ ✅ ✅
30.8% openrouter/openai/gpt-oss-120b ✅ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌ ✅ ❌ ✅ ❌
30.8% openrouter/qwen/qwen3-coder ✅ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ❌
30.8% openrouter/openai/gpt-oss-20b ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ✅ ❌ ✅ ✅ ❌
30.8% litellm/GLM-4.5-Air-FP8-dev ✅ ✅ ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌ ❌
0.0% openrouter/deepseek/deepseek-chat-v3-0324 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌