Evals
A landscape catalog of the platforms, frameworks, libraries, and benchmark suites used to evaluate large language models, LLM-based applications, and AI agents. The topic spans human-rated, LLM-as-a-judge, reference-based, reference-free, and benchmark-aligned approaches to measuring AI system quality. Tracked alongside the eval platforms are the canonical multi-task and code/agent benchmark suites (MMLU, HumanEval, GAIA, AgentBench, BIG-Bench) that establish public points of comparison.
More than an index entry, but the surface is still mostly links rather than artifacts — the cohort most likely to move a full band from modest, well-targeted work.
API Evangelist profiles Evals the way a machine reads it — 68 machine-readable artifacts across 20 APIs, pulled from the provider's own public surface and indexed so a developer, an analyst, or an AI agent can evaluate it against every other provider on the network.
Every provider in the network is reduced to the same set of machine-readable artifacts — OpenAPI contracts, event specifications, GraphQL schemas, runnable collections, pricing and rate-limit signals, security posture, OAuth scopes, and the agent surfaces (MCP servers and skills) that let software drive the API on its own. We profile them because the interface is the part of a company you can actually inspect: it is a truer signal of what a provider does than any marketing page. From those artifacts we compute the Kin Score — Evals scores 26.3/100 (emerging), with a separate agent-readiness read of 0/100 (human only). The full breakdown is below, followed by every artifact we hold — each card links through to its machine-readable definition on apis.io.
Kin Score
This is the API Evangelist rating — a single, repeatable read computed from the artifacts on this page. Green fill is points earned; the red track is points possible, so every bar shows earned-versus-possible at a glance.
Kin Score
How this is scored →scored 2026-07-27 · rubric v0.5
How we profile Evals
Each block below is one kind of artifact we hold for Evals. For each we say what it is and why it earns a place in the profile, then list every one we've indexed — capped at two rows, scroll within the panel for the rest.
APIs 20
Each API is captured as its own OpenAPI definition — every operation, parameter, and response. This is the single most useful machine-readable description of what an API does, and it's what lets us score, lint, mock, and generate against it without asking the provider for anything.
Individual APIs this provider publishes, each with its own machine-readable definition.
OpenAI Evals
OpenAI Evals is the open-source framework released by OpenAI for evaluating large language models and LLM-based systems. The README states "Evals provide a framework for evaluat...
Inspect AI
Inspect AI is an open-source framework for large language model evaluations developed and maintained by the UK AI Security Institute (UK AISI) and Meridian Labs. It supports tex...
Braintrust
Braintrust is a commercial evaluation platform that captures eval runs as immutable, comparable experiment snapshots. The product supports code-based scorers, built-in autoevals...
LangSmith Evaluation
LangSmith Evaluation is LangChain's evaluation framework for measuring application quality across the lifecycle. The docs describe evals as "a way to breakdown what 'good' looks...
Promptfoo
Promptfoo is an open-source CLI and library for evaluating and red-teaming LLM applications. The docs describe it as enabling "test-driven LLM development rather than trial-and-...
Helicone
Helicone is an open-source observability and monitoring platform for LLM applications. The homepage states "The world's fastest-growing AI companies rely on Helicone to route, d...
Patronus AI
Patronus AI is a frontier lab building evaluation infrastructure and Digital World Models for human-aligned AGI. Its evaluator models include Lynx (a hallucination-detection mod...
DeepEval (Confident AI)
DeepEval is an open-source LLM evaluation package, paired with Confident AI as the hosted observability/evals/monitoring tier. The docs call DeepEval "an open-source LLM eval pa...
Arize AI (Phoenix)
Arize AI provides an AI observability and evaluation platform centered on Arize AX (the commercial product) and Phoenix (open-source LLM tracing and evaluation). Phoenix runs LL...
Galileo
Galileo is an enterprise AI observability and evaluation engineering platform. The product line emphasizes "20+ built-in evaluators" spanning RAG, agents, safety, and security, ...
Humanloop
Humanloop was a development platform for LLM applications, describing itself as having been "the first development platform for LLM applications" and having "shaped industry sta...
TruLens
TruLens is an open-source evaluation and tracing platform for AI agents that helps developers "move from vibes to metrics." Its feedback-function library covers the RAG triad — ...
Weights and Biases Weave
W&B Weave is a platform for evaluating, monitoring, and iterating on AI agents and applications, started with "one line of code." Weave Evaluations enable visual comparison of r...
Ragas
Ragas is an open-source evaluation library focused on retrieval-augmented generation, described in its own docs as "a library that helps you move from 'vibe checks' to systemati...
MLflow LLM Evaluate
MLflow LLM evaluate extends MLflow's experiment tracking with mlflow.evaluate() support for LLM tasks. The API runs reference-based and reference-free metrics (toxicity, perplex...
MMLU Benchmark
MMLU (Measuring Massive Multitask Language Understanding) is a multiple-choice benchmark spanning 57 subjects from STEM and international law to nutrition and religion. It conta...
HumanEval Benchmark
HumanEval is OpenAI's evaluation harness for code-generation models, described in its README as "an evaluation harness for the HumanEval problem solving dataset described in the...
GAIA Benchmark
GAIA is "a benchmark for General AI Assistants," published in 2023 (arXiv 2311.12983). It tests general-purpose AI agent capability across reasoning, tool use, multi-modality, a...
AgentBench
AgentBench is the first benchmark designed to evaluate LLM-as-Agent across a diverse spectrum of environments. It bundles 8 environments — 5 newly created (Operating System, Dat...
BIG-Bench
The Beyond the Imitation Game Benchmark (BIG-Bench) is "a collaborative benchmark intended to probe large language models and extrapolate their future capabilities." It contains...
Scroll within the panel for all 20 ·
Features 10
The notable capabilities this provider advertises, captured as structured features so they can be searched and compared instead of read one landing page at a time.
Notable capabilities this provider offers.
LLM-as-a-Judge Scoring
A second LLM evaluates the output of the system-under-test, producing a numeric or categorical score and (optionally) a written rationale. The dominant scoring mode for free-for...
Reference-Based Scoring
Compares model output against a ground-truth expected answer using exact match, BLEU, ROUGE, embedding similarity, or task-specific equality (e.g. unit-test pass/fail). The nati...
Reference-Free Scoring
Assesses output quality without ground truth — toxicity, coherence, faithfulness against retrieved context, criterion adherence. Enables online (production-traffic) evaluation w...
Pairwise Comparison
A judge ranks two candidate outputs A vs B (or a tie). Useful when absolute scoring is hard but relative preference is reliable. Surfaced explicitly by LangSmith and used widely...
Benchmark-Aligned Evaluation
Runs the system-under-test against a standardized public dataset (MMLU, HumanEval, GAIA, AgentBench, BIG-Bench) to produce comparable, headline scores. The basis of model leader...
Human-Rated Scoring
Domain experts or end users provide thumbs-up/down, Likert ratings, or written critiques. Used as ground truth, as a judge-calibration signal, and as a final acceptance gate bef...
RAG Triad
Three feedback functions — groundedness, context relevance, answer relevance — codified by TruLens and widely adopted across Ragas, Phoenix, DeepEval, and LangSmith for evaluati...
Agent and Tool-Use Evaluation
Evaluating multi-step agent trajectories — did the agent pick the right tool, did the tool call succeed, did the final answer satisfy the goal. Supported by Inspect AI, Galileo,...
Online Production Monitoring
Eval scorers (typically reference-free LLM judges and Luna-style distilled evaluators) attach to live traffic via tracing/observability layers (Phoenix, Arize, Helicone, Galileo...
Red-Team / Safety Evaluation
Adversarial test suites probe jailbreaks, prompt injection, PII leakage, harmful content, and policy violations. First-class in Promptfoo, Patronus, Galileo, and Inspect AI's sa...
Scroll within the panel for all 10 ·
Semantic Vocabularies 1
JSON-LD contexts give the data shared meaning across APIs. We profile them because semantics are what let a machine reconcile 'customer' here with 'customer' somewhere else.
JSON-LD contexts and semantic vocabularies used across these APIs.
Evals Context
JSON-LDSpectral Rules 1
Governance rulesets we run against this provider's specs — the automated checks behind parts of the score. Profiling them makes the quality bar explicit and re-runnable, not a matter of opinion.
Evals API Rules
SPECTRALJSON Schema 6
Standalone JSON Schema definitions describe the data models behind the API. We profile them so the shapes are validatable on their own — useful long after a single request is forgotten.
Standalone JSON Schema definitions for this provider's data models.
JSON Structure 6
JSON Structure captures the data shapes in a form built for tooling — a complement to JSON Schema that keeps the model machine-legible.
JSON Structure definitions describing this provider's data shapes.
Evals Dataset Structure
JSON STRUCTUREEvals Eval Case Structure
JSON STRUCTUREEvals Eval Run Structure
JSON STRUCTUREEvals Eval Suite Structure
JSON STRUCTUREEvals Judge Structure
JSON STRUCTUREEvals Scorer Structure
JSON STRUCTUREExamples 9
Real request and response payloads are what turn a spec from abstract into obvious — and they're one of the twelve things an agent needs to call an API correctly on the first try.
Example request and response payloads for these APIs.
Evals Dataset Example
EXAMPLEEvals Eval Case Example
EXAMPLEEvals Eval Run Mmlu Example
EXAMPLEEvals Eval Suite Example
EXAMPLEEvals Judge Example
EXAMPLEEvals Scorer Example
EXAMPLEScroll within the panel for all 9 ·
Security Posture 1
Authentication, domain security, vulnerability disclosure, and trust-center signals — the evidence that a provider takes security seriously enough to document it. We profile it because you can't govern what you can't see.
Authentication, domain security, vulnerability disclosure, and trust-center signals.
Use Cases 8
What developers actually build with this provider — captured so the catalogue answers 'what is this for', not just 'what does this expose'.
What developers build with this provider.
Model Selection
Run candidate models (GPT-5, Claude 4.7, Gemini 3, open-weight) against a shared eval suite to choose the best fit for a specific application by quality, cost, and latency.
Prompt Engineering Iteration
Compare prompt variants in a matrix-style eval (Promptfoo, LangSmith experiments, Braintrust experiments) to pick the best prompt before shipping.
Regression Detection in CI/CD
Wire an eval suite into CI so a pull request that drops a key scorer below a threshold fails the build, preventing quality regressions from reaching production.
RAG Pipeline Tuning
Use RAG-triad scores (groundedness / context relevance / answer relevance) and faithfulness to tune chunking, embedding, reranking, and prompt choices.
Agent Trajectory Quality
Score multi-step agent runs on tool-selection correctness, step efficiency, and final-answer faithfulness — the core measurement for production agentic apps.
Hallucination and Safety Guardrails
Deploy dedicated judge models (Lynx, GLIDER, Luna) to flag hallucinations, toxic content, PII leakage, and policy violations in real time.
Frontier Capability and Safety Assessment
Independent labs (UK AISI, US AISI) run capability and safety evaluations on frontier models before release, using frameworks like Inspect AI.
Public Leaderboard Reporting
Submit a model's scores against MMLU, HumanEval, GAIA, AgentBench, and BIG-Bench to position it on community leaderboards and back marketing claims with reproducible numbers.
Scroll within the panel for all 8 ·
Integrations 6
Pre-built integrations with other platforms tell you where this provider already fits in a stack.
Pre-built integrations with other platforms and tools.
OpenTelemetry
Phoenix, TruLens, Weave, and most modern eval platforms ingest LLM traces via OpenTelemetry, making eval a layer on top of standard observability.
LangChain / LangGraph
LangSmith is the native eval tier for LangChain/LangGraph apps; most other platforms also integrate.
LlamaIndex
Ragas, DeepEval, and Phoenix integrate directly with LlamaIndex for RAG evaluation.
Hugging Face Datasets
MMLU, HumanEval, and GAIA are distributed as Hugging Face datasets and consumed by every eval framework.
CI/CD (GitHub Actions, etc.)
Braintrust, Promptfoo, LangSmith, and DeepEval ship CI integrations to fail builds on regression.
Snowflake
OpenAI Evals can log eval results to Snowflake; TruLens (Truera) is now part of Snowflake.
Resources
Every other property we hold for Evals — documentation, portals, status pages, policies, and corporate surface — grouped by the job it does, following the integrator's arc from getting started to running in production.
Documentation 6
Reference material describing how the API behaves
Design & Contract 3
Pagination, idempotency, versioning, errors, and events
Build 1
SDKs, sample code, and the tooling you integrate with
Access & Security 1
Authentication, authorization, and security posture
← All providers · Data indexed from github.com/api-evangelist/evals · machine-readable index on apis.io