LLM Evaluation

← Back to main page

Run of 2026-01-18 15:00:50

16
Models Tested
13
Total Tasks
42.8%
Overall Success Rate
89/208
Passed Combinations

Details

Score Model task10_multiple_tests task11_relationship_classifier task12_need_reply task13_meeting_action_items task14_graph_money_distribution task1_file_list task2_fix_python_syntax task4_web_fetch task5_dedup_contact task6_config_merger task7_log_parser task8_regex_extraction task9_cpp_footguns
92.3% openrouter/google/gemini-3-pro-preview ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅
92.3% openrouter/anthropic/claude-opus-4.5 ✅ ✅ ✅ ✅ ❌ ✅ ✅ ✅ ✅ ✅ ✅ ✅ ✅
69.2% openrouter/google/gemini-2.5-pro ✅ ✅ ✅ ❌ ❌ ❌ ✅ ❌ ✅ ✅ ✅ ✅ ✅
61.5% openrouter/anthropic/claude-haiku-4.5 ✅ ✅ ❌ ❌ ❌ ❌ ✅ ❌ ✅ ✅ ✅ ✅ ✅
61.5% openrouter/anthropic/claude-sonnet-4.5 ✅ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ✅ ✅ ✅ ✅ ✅
53.8% openrouter/deepseek/deepseek-v3.1-terminus ✅ ✅ ❌ ❌ ❌ ✅ ✅ ❌ ✅ ✅ ❌ ✅ ❌
38.5% openrouter/google/gemini-2.5-flash-preview-09-2025 ✅ ❌ ❌ ❌ ✅ ❌ ✅ ❌ ❌ ❌ ❌ ✅ ✅
38.5% openrouter/google/gemini-2.5-flash-lite-preview-09-2025 ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ ✅ ❌ ❌ ✅ ❌
38.5% openrouter/x-ai/grok-code-fast-1 ❌ ✅ ✅ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ✅ ❌ ❌
30.8% openrouter/x-ai/grok-3-mini ✅ ❌ ✅ ❌ ❌ ✅ ❌ ❌ ❌ ✅ ❌ ❌ ❌
30.8% litellm/GLM-4.5-Air-FP8-dev ✅ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌ ❌ ✅ ✅ ❌
23.1% openrouter/openai/gpt-oss-120b ✅ ❌ ❌ ❌ ✅ ✅ ❌ ❌ ❌ ❌ ❌ ❌ ❌
23.1% openrouter/qwen/qwen3-coder ✅ ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ✅ ❌ ❌ ❌
15.4% openrouter/openai/gpt-4o-mini ✅ ❌ ❌ ❌ ❌ ❌ ✅ ❌ ❌ ❌ ❌ ❌ ❌
15.4% openrouter/openai/gpt-oss-20b ✅ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ✅ ❌
0.0% openrouter/deepseek/deepseek-chat-v3-0324 ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌ ❌