Today we are releasing ACUMEN, a composite benchmark for evaluating large language models across three dimensions: intelligence, agentic task completion, and inference efficiency.
Most existing benchmarks evaluate these dimensions in isolation. A model that reasons well but requires excessive compute to do so is not strictly better for deployment. A model that scores highly on static tasks but fails in dynamic tool environments is not ready for real-world use. ACUMEN is designed to surface these tradeoffs as a first-class measurement.
The benchmark is organized into three domains. ACUMEN-I evaluates formal reasoning, mathematical problem solving, code comprehension, and long-context synthesis. ACUMEN-A evaluates goal-directed task completion in dynamic tool environments, including multi-step planning, tool use, error recovery, and adversarial conditions. ACUMEN-E measures inference efficiency through a Quality-per-FLOP metric, time to first token, throughput, and memory footprint on standardized hardware.
Scores are combined into a single composite using a weighted formula that reflects the relative importance of each domain in real deployment contexts.
We are publishing the full methodology, evaluation harness, and agentic sandbox as open source. The test dataset remains internal to prevent contamination. Organizations wishing to evaluate their models can submit via the leaderboard page.
SAGE Actus 2.4 and SAGE Celer models are included in the initial results.
- View leaderboard → /acumen
- Read methodology → /acumen/methodology
- GitHub: acumen-eval · acumen-sandbox