VigilSAR Public Leaderboard Shows Defense-ISR LLM Benchmark Results

VigilSAR Defense LLM Benchmark — The public benchmark page — aggregate results public, task set private. Source: vigilsar.com

VigilSAR, a specialized defense-ISR software platform, has published its latest public LLM leaderboard, offering a rare glimpse into how language models perform in intelligence-surveillance-reconnaissance tasks. Unlike typical benchmarks that focus on trivia or general knowledge, this one emphasizes reasoning, reporting, and restraint, which are critical for accurate analysis in sensitive environments.

The setup involved testing 14 models across 300 tasks with scores recorded on July 17, 2026. Importantly, the public results are fully available, but the actual task set remains private. This deliberate choice prevents models from training on the exact tasks, maintaining a high level of integrity and preventing data contamination. A private held-out set exists for validation, with the gap between public and private scores published per model, serving as an indicator of memorization or overfitting.

In the current standings, claude-fable-5 leads with a band score of 67.77. A new entrant, Moonshot’s Kimi K3, debuts impressively at 64.65, placing it in Band B, ahead of many GPT and Gemini models. The leaderboard categorizes models into bands based on their confidence intervals, rather than providing a strict rank, reflecting the uncertainty inherent in these evaluations. Notably, one locally deployable model is scored as ‘sovereign-deployable’, incorporating real-world deployment considerations into its score.

The purpose of this evaluation is to establish evidence-based comparisons, emphasizing that vendor claims are not enough. The operators built the evaluation to determine which models are viable for their own defense-ISR needs and to rank models they use without bias, all outside of vendor influence. This approach underscores the importance of transparency and honesty in AI benchmarking for critical applications.

Additional honesty features include the use of bands instead of precise ranks, published confidence intervals, and held-out score gaps to provide context on model performance. A public leaderboard offers ongoing updates, and the site publishes per-model economics like cost-per-correct-answer, giving a comprehensive view of both capabilities and practical deployment factors.

For tech enthusiasts, understanding the significance of this benchmark means recognizing that task privacy and deployment considerations are central to defense-ISR AI development. The recent debut of Kimi K3 ahead of GPT and Gemini models highlights that performance in specialized tasks can diverge from general-purpose language models, proving the value of targeted evaluation. This effort exemplifies a transparent, practical approach to AI benchmarking in high-stakes environments.

As the field evolves, VigilSAR’s approach of combining private task sets with open scoring, confidence bands, and deployment-relevant scoring offers a model for responsible AI evaluation. Interested readers can explore the detailed standings and methodology at the public leaderboard, and learn more about VigilSAR’s mission at VigilSAR.