GAIA predicted score
45.34%
Eugeny 5.1 Vision · PACE C100
Measured,
not marketed.
A complete, reproducible proxy evaluation of Eugeny 5.1’s agentic capability—published with the uncomfortable parts intact.
SWE-bench Verified predicted score
68.89%
SWE-bench Multimodal predicted score
16.93%
SWTBench predicted score
59.90%
Why is multimodal low?
The model transported images correctly, but broad visual understanding was never established by the serving qualification. The strongest direct evidence is the PACE visual subset: weak academic-image reasoning and web-screenshot grounding, with only middling visual-puzzle performance.
Eugeny 5.1 is primarily a coding and text reasoning model. Its vision derivative passed a narrow single-image serving gate; that proves the interface works, not that visual reasoning is competitive.
MMMU
18.75%
VisualWebBench
19.20%
VisualPuzzles
48.15%
Estimate with caution.
PACE’s multimodal target predictor is the least trustworthy part of this result. On its reference models, its leave-one-model-out diagnostics show almost no useful rank or linear correlation. The 16.93% estimate is directionally consistent with the weak visual proxies, but it should not be treated as Eugeny’s directly measured SWE-bench Multimodal ceiling.
LOOCV MAE
6.69 pts
Spearman
−0.131
Pearson
0.016
R²
−0.245
What would settle it?
A direct, fidelity-preserving SWE-bench Multimodal run. Until then, 16.93% is a proxy-derived estimate with poor target calibration—not a direct benchmark score.
Run facts.
The full selection finished after one transient LeetCode submission failure was recovered. The identical credentials succeeded on the first retry, ruling out persistent token expiry. Final coverage is complete.
- Model
- Eugeny 5.1 Vision, BF16, unquantized
- Sampling
- Temperature 0.0 · deterministic greedy decoding
- Coverage
- 412 / 412 selection rows · 385 unique standardized keys
- Context
- 262,144 tokens served maximum
- Media scope
- At most one image per prompt; multi-image reasoning not qualified
- Runtime
- vLLM 0.25.0 · compiled CUDA graphs
- PACE source
dc2ef80e00addd519e7d8479f875cc3ecb46c6cb- Evaluator
sha256:c94bda538bb5ca1fdd53fecc0e1899e63184f40a978110060d1818c255e91994- Recovery
- One DebugBench row retried and scored 0.0; predictors then rerun on complete coverage
- Published
- 16 July 2026 · Europe/Rome