Benchmark
Self-hosted and cloud vision models, 1,200 calls compared
For my thesis, Hesperus, I needed to pick the vision model that checks each photo the camera takes. I ran the same scenes through six setups, three cloud and two self-hosted plus one control, and had an LLM judge score the answers, with a second judge from another provider as a check. The raw data and the script are in the repo.
Result
What I found
A 30B model I hosted myself scored about as well as the best cloud model. The judge gave it 6.05 out of 9 against 6.60, which is inside the noise of the measurement.
It answered in about 0.77 seconds. The cloud models took 3.6 to 7.8 seconds for the same scenes, and they charge per call.
This is one workload, so it says nothing certain about yours. It was enough for me to choose the self-hosted model for Hesperus.
Numbers
The main results
Three cloud models, the same cloud model again with extended thinking off, and two self-hosted models, on the same scenes.
Method
How I ran it
Hesperus is a camera on an STM32 board that sleeps most of the time. An agent decides when it should wake and take a photo, and a vision model checks each photo against what the user asked for. The benchmark covers both steps, planning the photos and checking them.
Every setup saw the same scenes, many times over, so a single lucky or unlucky call couldn't decide anything. I logged the latency and cost of each call.
Claude Sonnet 4.6 scored the answers from 0 to 9. Gemini 3 Flash scored them again as a check, and the two ranked the models almost the same way, with a Spearman correlation of 0.83. The full results are in results/findings_v3.md in the repo, next to the runner and the raw rows.
Details
Two things I didn't expect
Turning extended thinking on made the cloud model slower and more expensive, and it didn't score any better on this task.
The smaller self-hosted model, a 3B, was too small for the job. It scored lowest and couldn't make tool calls at all, so it was never an option for an agent.