Score
Rate one response against a weighted rubric, from 0 to 10.
Open evaluation templates
Evaluate AI
with AI.
A provider-agnostic starting point for scoring responses, comparing models, evaluating RAG, testing agents, and adding reliable quality gates to your AI applications.
quickstart.pyPython
from llmasjudge import Judge judge = Judge( model="your-judge-model", criteria=[ "correctness", "relevance", "groundedness" ] ) result = judge.evaluate( question=question, answer=model_response, reference=expected_answer ) print(result.score) # 8.7 print(result.passed) # True
Four core patterns
One judge. Many jobs.
Start with the evaluation shape that fits your workflow. Every pattern returns structured evidence you can inspect, store, and compare.
Pass / Fail
Turn an evaluation threshold into a CI or regression quality gate.
Pairwise
Compare response A and B while reducing position and style bias.
Multi-judge
Aggregate several judges for harder or higher-risk decisions.
Input
Judge
Rubric
Evidence
Verdict
Runnable starter
Download. Run. Judge.
A complete Python project with a reusable Judge class, weighted rubrics, validated JSON output, tests, and an optional real-model adapter.
llmasjudge-starter.zip
Works before you add an API key.
The built-in mock model lets developers understand the result contract immediately. When ready, set three environment variables to connect an OpenAI-compatible judge model.
Zero dependenciesMIT licenseTests includedOpenAI-compatible
Download project ↓$ unzip llmasjudge-starter.zip $ cd llmasjudge-starter $ python examples/basic.py Running with MockLLMClient... { "overall_score": 8.6, "verdict": "PASS" }
Interactive example
See the contract.
This browser demo shows the structure of a judge result. Connect the template to your preferred model to run real evaluations.
Overall score
8.7/10
Judge reasoningEvidence
The response is correct, concise, and directly explains the retrieval and generation steps. It could be slightly more complete by mentioning improved grounding and reduced hallucination risk.
Copy. Adapt. Ship.
Generic templates.
The interface stays stable while the provider adapter changes. Keep prompts, schemas, scoring, and model calls separated.
judge.pyPython
Build it responsibly
A practical judge checklist.
LLM judges are useful evaluators, not unquestionable truth machines. Treat their decisions as measurements with known limits.
Write a specific rubric
Define what good means for the task. Separate correctness, relevance, completeness, groundedness, safety, or tone instead of asking for a vague quality score.
Require structured output
Use a validated JSON schema, bounded scores, short reasoning, and evidence fields. Retry invalid responses and retain the raw output for debugging.
Control common bias
Randomize pairwise order, hide model identity, avoid rewarding length, and test whether your judge prefers its own writing style.
Calibrate with humans
Build a labeled test set, compare judge and human ratings, inspect disagreement, and choose thresholds from real errors rather than intuition.
Use more than one signal
Combine judge scores with deterministic checks, citations, task success, latency, cost, and human review for consequential decisions.
Track versions
Record the judge model, prompt, rubric, temperature, timestamp, and dataset version so evaluation results remain reproducible.
LLMasJudge.com
Build evaluation into the product, not after it.
Download the project. Run the demo. Replace the adapter. Add your rubric. Measure what changes.
Download the starter project ↓