Provider adapters
Add native examples for Anthropic, Gemini, Bedrock, Azure OpenAI, Ollama, and other models.
Public open-source project
Use it. Change it. Build on it.
The LLMasJudge Python starter is public and released under the permissive MIT License. Use it for personal work, research, internal tools, or commercial products.
LicenseMITSimple and permissive
What the MIT License allows
You may use, copy, modify, merge, publish, distribute, sublicense, and sell copies of the software. Keep the copyright and permission notice with substantial copies of the project.
The software is provided without warranty. Review it for your own production, privacy, security, and regulatory needs.
Inside the download
A small project with real structure.
llmasjudge-starter/
├── llmasjudge/
│ ├── judge.py
│ ├── clients.py
│ └── prompts.py
├── examples/basic.py
├── tests/test_judge.py
├── README.md
├── CONTRIBUTING.md
├── SECURITY.md
├── CHANGELOG.md
└── LICENSE
Ways to contribute
Useful work to take on next.
Evaluation patterns
Contribute pairwise, multi-judge, RAG, agent trajectory, safety, and regression evaluators.
Calibration tools
Help compare judge decisions with human labels and measure agreement, bias, and reliability.
Project standards
Open does not mean careless.
Contributions should include clear documentation and tests when behavior changes. Never submit API keys, personal data, customer information, proprietary prompts, or confidential evaluation datasets.
Creator
Maintained by Sarath Surya.
Questions, ideas, or collaboration proposals? Connect with Sarath on LinkedIn.
LinkedIn ↗