Scorer
Says how good an answer is.
in code
deterministic logic
LLM as judge
a model grades the answer
Offline
A fixed dataset.
Every change, one table.
dataset
a change
model
prompt
logic
score
evaluation
check a change before it ships
compare quality against cost on the same table
every production failure becomes a case, so it never regresses twice
Online
A sample of real traffic.
Scored as it happens.
real traffic
sample
score
alert
watch quality drift on real traffic, it alerts, never blocks
catch the change you did not make: the provider swapping the model
find the cases the dataset is missing