Back

Benchmark

Self-hosted and cloud vision models, 1,200 calls compared

For my thesis, Hesperus, I needed to pick the vision model that checks each photo the camera takes. I ran the same scenes through six setups, three cloud and two self-hosted plus one control, and had an LLM judge score the answers, with a second judge from another provider as a check. The raw data and the script are in the repo.

Calls
1,200 across six setups
Judges
Claude, checked by Gemini
Run
May 2026
Data
Public
Published

Result

What I found

A 30B model I hosted myself scored about as well as the best cloud model. The judge gave it 6.05 out of 9 against 6.60, which is inside the noise of the measurement.

It answered in about 0.77 seconds. The cloud models took 3.6 to 7.8 seconds for the same scenes, and they charge per call.

This is one workload, so it says nothing certain about yours. It was enough for me to choose the self-hosted model for Hesperus.

Numbers

The main results

Three cloud models, the same cloud model again with extended thinking off, and two self-hosted models, on the same scenes.

0.77smedian answer time, self-hosted 30B model
3.6 to 7.8smedian answer time, cloud models
6.05 / 6.60quality out of 9, self-hosted against the best cloud model
1,200calls in total

Method

How I ran it

Hesperus is a camera on an STM32 board that sleeps most of the time. An agent decides when it should wake and take a photo, and a vision model checks each photo against what the user asked for. The benchmark covers both steps, planning the photos and checking them.

Every setup saw the same scenes, many times over, so a single lucky or unlucky call couldn't decide anything. I logged the latency and cost of each call.

Claude Sonnet 4.6 scored the answers from 0 to 9. Gemini 3 Flash scored them again as a check, and the two ranked the models almost the same way, with a Spearman correlation of 0.83. The full results are in results/findings_v3.md in the repo, next to the runner and the raw rows.

Details

Two things I didn't expect

Turning extended thinking on made the cloud model slower and more expensive, and it didn't score any better on this task.

The smaller self-hosted model, a 3B, was too small for the job. It scored lowest and couldn't make tool calls at all, so it was never an option for an agent.