LLM Evaluation

← Back to main page

Run of 2026-02-21 15:01:07

16
Models Tested
13
Total Tasks
51.9%
Overall Success Rate
108/208
Passed Combinations

Details

Score Model task10_multiple_tests task11_relationship_classifier task12_need_reply task13_meeting_action_items task14_graph_money_distribution task1_file_list task2_fix_python_syntax task4_web_fetch task5_dedup_contact task6_config_merger task7_log_parser task8_regex_extraction task9_cpp_footguns
100.0% openrouter/anthropic/claude-opus-4.5 ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
92.3% litellm/Kimi-K2.5-sandbox ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅
76.9% openrouter/x-ai/grok-code-fast-1 ✅ ✅ ✅ ❌ ❌ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅
69.2% openrouter/google/gemini-3-pro-preview ✅ ✅ ❌ ❌ ✅ ❌ ✅ ❌ ✅ ✅ ✅ ✅ ✅
69.2% openrouter/deepseek/deepseek-v3.1-terminus ✅ ✅ ✅ ❌ ❌ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ❌
69.2% openrouter/anthropic/claude-sonnet-4.5 ✅ ✅ ❌ ❌ ❌ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅
61.5% openrouter/anthropic/claude-haiku-4.5 ✅ ✅ ❌ ❌ ❌ ❌ ✅ ❌ ✅ ✅ ✅ ✅ ✅
53.8% openrouter/google/gemini-2.5-pro ✅ ✅ ✅ ❌ ✅ ✅ ✅ ❌ ✅ ❌ ❌ ❌ ❌
46.2% openrouter/google/gemini-2.5-flash-lite-preview-09-2025 ✅ ✅ ❌ ❌ ❌ ✅ ✅ ❌ ✅ ❌ ❌ ✅ ❌
38.5% openrouter/x-ai/grok-3-mini ❌ ✅ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ✅ ✅ ❌ ❌
38.5% openrouter/openai/gpt-oss-20b ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ ✅ ❌ ❌ ✅ ❌
38.5% litellm/GLM-4.5-Air-FP8-dev ✅ ❌ ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ✅ ✅ ❌
30.8% openrouter/openai/gpt-4o-mini ✅ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ❌ ❌ ❌ ✅ ❌
23.1% openrouter/openai/gpt-oss-120b ✅ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ✅ ✅ ❌
23.1% openrouter/qwen/qwen3-coder ✅ ❌ ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌ ❌
0.0% openrouter/deepseek/deepseek-chat-v3-0324 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌