Ragas

Ragas is an open-source evaluation toolkit for Large Language Model applications, with particular depth on Retrieval Augmented Generation (RAG) and agentic systems. Originally created under the Exploding Gradients organization on GitHub and now maintained by Vibrant Labs AI, Ragas is a Python library distributed on PyPI under the Apache 2.0 license. It moves teams from informal "vibe checks" to systematic evaluation loops by providing objective LLM-based and traditional metrics, automated test dataset generation, experiment tracking, and integrations with the broader LLM ecosystem including LangChain, LlamaIndex, OpenAI, Anthropic, and popular observability platforms. Ragas exposes a metrics library covering faithfulness, response relevancy, context precision and recall, factual correctness, semantic similarity, agent tool-use accuracy, SQL equivalence, Nvidia-defined RAG metrics, and general-purpose rubric scoring. The project ships a CLI (`ragas`) with quickstart templates such as `rag_eval`, and is consumed primarily as a `pip install ragas` library rather than as a hosted API service. Ragas is widely cited as a default evaluation harness for RAG applications and has grown a substantial community on GitHub and Discord.

Kin Score

This is the API Evangelist rating — a single, repeatable read computed from the artifacts on this page. Green fill is points earned; the red track is points possible, so every bar shows earned-versus-possible at a glance.

How we profile Ragas

Each block below is one kind of artifact we hold for Ragas. For each we say what it is and why it earns a place in the profile, then list every one we've indexed — capped at two rows, scroll within the panel for the rest.

APIs 1

Each API is captured as its own OpenAPI definition — every operation, parameter, and response. This is the single most useful machine-readable description of what an API does, and it's what lets us score, lint, mock, and generate against it without asking the provider for anything.

Individual APIs this provider publishes, each with its own machine-readable definition.

Ragas Python Library

The Ragas Python library is the primary surface of the project, installed via `pip install ragas` and imported as `ragas`. It exposes evaluation entry points (`ragas.evaluate`),...

Features 10

The notable capabilities this provider advertises, captured as structured features so they can be searched and compared instead of read one landing page at a time.

Notable capabilities this provider offers.

RAG Evaluation Metrics

Faithfulness, Response Relevancy, Context Precision, Context Recall, Context Entities Recall, and Noise Sensitivity for retrieval augmented generation pipelines.

Agent and Tool-Use Metrics

Topic Adherence, Tool Call Accuracy, Tool Call F1, and Agent Goal Accuracy for evaluating multi-step agentic systems.

Natural Language Comparison

Factual Correctness, Semantic Similarity, BLEU, ROUGE, CHRF, Exact Match, and String Presence metrics for output comparison.

SQL Evaluation

Execution-based Datacompy Score and SQL Query Equivalence metrics for text-to-SQL applications.

General Purpose Scoring

Aspect Critic, Simple Criteria Scoring, Rubrics-based scoring, and instance-specific rubrics for custom evaluation criteria.

Nvidia Metrics

Answer Accuracy, Context Relevance, and Response Groundedness metrics contributed by Nvidia for RAG quality.

Test Data Generation

Automated synthesis of diverse test datasets covering single-hop, multi-hop, and abstract query types over user knowledge bases.

Experiments

Experiment-first workflow comparing prompts, models, and configurations across datasets with iterative result tracking.

Custom Metrics

DiscreteMetric and decorator-based APIs for defining LLM-judge and rule-based custom evaluation metrics.

CLI Quickstart Templates

The `ragas quickstart` command scaffolds evaluation projects including the `rag_eval` template for RAG systems.

Scroll within the panel for all 10 ·

Security Posture 1

Authentication, domain security, vulnerability disclosure, and trust-center signals — the evidence that a provider takes security seriously enough to document it. We profile it because you can't govern what you can't see.

Authentication, domain security, vulnerability disclosure, and trust-center signals.

Ragas Ai Domain Security

TLSv1.3 · HSTS · DMARC

SECURITY

Use Cases 6

What developers actually build with this provider — captured so the catalogue answers 'what is this for', not just 'what does this expose'.

What developers build with this provider.

RAG Pipeline Evaluation

Scoring retrieval and generation quality in RAG applications across faithfulness, relevance, and context fidelity.

Agent Evaluation

Measuring tool-call correctness, goal completion, and topic adherence in multi-step LLM agents.

Regression Testing in CI

Running Ragas metrics in CI pipelines to detect quality regressions across prompt, model, and configuration changes.

Model and Prompt Selection

Comparing candidate models and prompt variants on a fixed dataset using Ragas experiments.

Synthetic Test Set Generation

Generating diverse evaluation datasets from a knowledge base for systematic LLM testing.

Text-to-SQL Evaluation

Validating generated SQL against reference queries using execution and structural equivalence metrics.

Integrations 9

Pre-built integrations with other platforms tell you where this provider already fits in a stack.

Pre-built integrations with other platforms and tools.

LangChain

Native integration for evaluating LangChain chains, retrievers, and agents using Ragas metrics.

LlamaIndex

Integration for evaluating LlamaIndex RAG pipelines and query engines.

OpenAI

Default LLM judge backend uses OpenAI models such as GPT-4 class judges.

Anthropic

Anthropic Claude models supported as LLM judges via the LangChain LLM abstraction.

Hugging Face

Support for Hugging Face embeddings and models as judges, plus dataset interop via the `datasets` library.

LangSmith

Result tracking and trace inspection via LangSmith observability.

Arize Phoenix

Observability integration for tracing Ragas evaluations alongside production LLM traffic.

Helicone

LLM cost and trace observability for Ragas-driven evaluations.

Pandas

Datasets and evaluation results are exposed as pandas DataFrames for analysis.

Scroll within the panel for all 9 ·

Resources

Every other property we hold for Ragas — documentation, portals, status pages, policies, and corporate surface — grouped by the job it does, following the integrator's arc from getting started to running in production.

← All providers · Data indexed from github.com/api-evangelist/ragas-ai · machine-readable index on apis.io