BenchLLM
Open-source Python tool to evaluate LLM apps with test suites, reports and a CI-friendly CLI, built by V7.
What it does
BenchLLM is an open-source library for evaluating LLM-powered applications. Engineers build test suites in JSON or YAML, run automated, interactive or custom evaluations, generate quality reports, and wire checks into CI/CD to catch regressions.
Core features
Test suites for LLM outputs
Automated/interactive/custom eval strategies
CLI for CI/CD pipelines
Quality and regression reports
Works with OpenAI, LangChain, any API
Semantic evaluators
Best for
→Evaluate LLM app responses
→Catch hallucinations and regressions
→Automate model testing in CI
→Compare model quality over time