Need help with your APIs? I offer API discovery, governance & evangelism services. Explore services →
API Evangelist API Evangelist
Discovery
Learnings
Guidance
Toolbox
Alignment
API Evangelist LLC
Apache Nutch website screenshot

Apache Nutch

Apache Nutch is a highly extensible and scalable open-source web crawler software project built on Apache Hadoop data structures for batch processing. It provides a pluggable architecture supporting custom parse filters, scoring filters, index writers, and protocol implementations. Nutch integrates with Apache Solr and Elasticsearch for full-text search and exposes a REST API for managing crawl jobs, configurations, seed lists, and database queries. Governed by the Apache Software Foundation under the Apache License 2.0.

agent ready

Solid contracts, transparent operations, and an easy start — typically complete on four or five facets with one clear soft spot.

Kin Score

API Evangelist profiles Apache Nutch the way a machine reads it — 86 machine-readable artifacts across 7 APIs, pulled from the provider's own public surface and indexed so a developer, an analyst, or an AI agent can evaluate it against every other provider on the network.

Every provider in the network is reduced to the same set of machine-readable artifacts — OpenAPI contracts, event specifications, GraphQL schemas, runnable collections, pricing and rate-limit signals, security posture, OAuth scopes, and the agent surfaces (MCP servers and skills) that let software drive the API on its own. We profile them because the interface is the part of a company you can actually inspect: it is a truer signal of what a provider does than any marketing page. From those artifacts we compute the Kin Score — Apache Nutch scores 63.3/100 (strong), with a separate agent-readiness read of 48/100 (agent ready). The full breakdown is below, followed by every artifact we hold — each card links through to its machine-readable definition on apis.io.

Kin Score

This is the API Evangelist rating — a single, repeatable read computed from the artifacts on this page. Green fill is points earned; the red track is points possible, so every bar shows earned-versus-possible at a glance.

Kin Score Kin Score How this is scored →
scored 2026-07-27 · rubric v0.5
Composite quality — 63.3/100 · strong
Contract Quality 19.9 / 25
Developer Ergonomics 7.4 / 20
Commercial Clarity 10.0 / 20
Operational Transparency 6.8 / 13
Governance 10.4 / 12
Discoverability 8.8 / 10
Agent readiness — 48/100 · agent ready
Machine-Readable Contract 18 / 18
Agentic Access Contract 15 / 15
MCP Server 0 / 12
Machine-Readable Auth 10 / 10
Idempotency 0 / 9
Stable Error Semantics 0 / 8
Request/Response Examples 0 / 7
Rate-Limit Signaling 7 / 7
Typed Event Surface 0 / 6
Agent Skills 0 / 5
Well-Known Catalog 0 / 4
Consent & Bot Identity 0 / 3

How we profile Apache Nutch

Each block below is one kind of artifact we hold for Apache Nutch. For each we say what it is and why it earns a place in the profile, then list every one we've indexed — capped at two rows, scroll within the panel for the rest.

APIs 7

Each API is captured as its own OpenAPI definition — every operation, parameter, and response. This is the single most useful machine-readable description of what an API does, and it's what lets us score, lint, mock, and generate against it without asking the provider for anything.

Individual APIs this provider publishes, each with its own machine-readable definition.

Apache Nutch Admin API

Server administration operations

Apache Nutch Configuration API

Manage Nutch configurations

Apache Nutch Database API

Query the CrawlDB and FetchDB

Apache Nutch Job API

Manage crawl jobs

Apache Nutch Reader API

Read sequence files and webgraph data

Apache Nutch Seed API

Manage seed URL lists

Apache Nutch Services API

Auxiliary service operations such as CommonCrawl data dumps

Scroll within the panel for all 7 ·

Pricing Plans 1

Pricing is part of the interface. Machine-readable plans tell you what a tier costs and includes before you commit — one of the six things the Kin Score reads for commercial clarity.

Published pricing tiers and plan structures.

Rate Limits 1

Rate limits are the difference between a demo that works and a production integration that doesn't fall over. Publishing them is an operational-transparency signal — and a hard requirement for any agent that plans its own throughput.

Documented rate limits and quota policies.

Apache Nutch Rate Limits

5 limits

RATE LIMITS

FinOps 1

Cost, billing, and metering signals let a buyer model the financial operations of an API before it's live. We profile them for the same reason we profile pricing: the money is part of the contract.

Cost, billing, and metering signals for API financial operations.

Features 10

The notable capabilities this provider advertises, captured as structured features so they can be searched and compared instead of read one landing page at a time.

Notable capabilities this provider offers.

Scalable Batch Crawling

Leverages Apache Hadoop data structures for distributed, large-scale web crawling batch processing.

Pluggable Architecture

Extensible plugin system supporting custom parse filters, scoring filters, index writers, protocol plugins, and URL filters.

REST API for Crawl Management

Full REST API for managing crawl jobs, configurations, seed lists, CrawlDB/FetchDB queries, and sequence file readers.

Full-Text Search Integration

Built-in index writers for Apache Solr and Elasticsearch to enable full-text search over crawled content.

Apache Tika Parsing

Uses Apache Tika for parsing a wide variety of document formats during the crawl pipeline.

Duplicate Detection

Built-in deduplication support to identify and remove duplicate content from the crawl database and search index.

Configurable URL Filtering

Regex-based and custom URL filter plugins to control crawl scope and exclusions.

Incremental Crawling

Supports multi-round incremental crawling workflows to keep the crawl database fresh.

CommonCrawl Export

Service operations for exporting crawl data in CommonCrawl-compatible formats.

HTTP Authentication Support

Configurable HTTP authentication schemes for crawling password-protected sites.

Scroll within the panel for all 10 ·

Semantic Vocabularies 1

JSON-LD contexts give the data shared meaning across APIs. We profile them because semantics are what let a machine reconcile 'customer' here with 'customer' somewhere else.

JSON-LD contexts and semantic vocabularies used across these APIs.

Apache Nutch Context

16 classes · 35 properties

JSON-LD

Spectral Rules 2

Governance rulesets we run against this provider's specs — the automated checks behind parts of the score. Profiling them makes the quality bar explicit and re-runnable, not a matter of opinion.

Apache Nutch API Rules

5 rules · 3 warnings

SPECTRAL

Apache Nutch API Rules

39 rules · 12 errors · 24 warnings

SPECTRAL

JSON Schema 16

Standalone JSON Schema definitions describe the data models behind the API. We profile them so the shapes are validatable on their own — useful long after a single request is forgotten.

Standalone JSON Schema definitions for this provider's data models.

ChildNode

2 properties

JSON SCHEMA

DbQuery

4 properties

JSON SCHEMA

FetchNodeDbInfo

4 properties

JSON SCHEMA

JobConfig

5 properties

JSON SCHEMA

JobInfo

8 properties

JSON SCHEMA

JobType

0 properties

JSON SCHEMA

LinkSchema

6 properties

JSON SCHEMA

NodeSchema

6 properties

JSON SCHEMA

NutchConfig

3 properties

JSON SCHEMA

NutchServerInfo

4 properties

JSON SCHEMA

ReaderConfig

1 properties

JSON SCHEMA

SeedList

4 properties

JSON SCHEMA

SeedUrl

2 properties

JSON SCHEMA

ServiceConfig

3 properties

JSON SCHEMA

ServiceInfo

1 properties

JSON SCHEMA

State

0 properties

JSON SCHEMA

Scroll within the panel for all 16 ·

JSON Structure 16

JSON Structure captures the data shapes in a form built for tooling — a complement to JSON Schema that keeps the model machine-legible.

JSON Structure definitions describing this provider's data shapes.

Apache Nutch Child Node Structure

2 properties

JSON STRUCTURE

Apache Nutch Db Query Structure

4 properties

JSON STRUCTURE

Apache Nutch Fetch Node Db Info Structure

4 properties

JSON STRUCTURE

Apache Nutch Job Config Structure

5 properties

JSON STRUCTURE

Apache Nutch Job Info Structure

8 properties

JSON STRUCTURE

Apache Nutch Job Type Structure

0 properties

JSON STRUCTURE

Apache Nutch Link Schema Structure

6 properties

JSON STRUCTURE

Apache Nutch Node Schema Structure

6 properties

JSON STRUCTURE

Apache Nutch Nutch Config Structure

3 properties

JSON STRUCTURE

Apache Nutch Nutch Server Info Structure

4 properties

JSON STRUCTURE

Apache Nutch Reader Config Structure

1 properties

JSON STRUCTURE

Apache Nutch Seed List Structure

4 properties

JSON STRUCTURE

Apache Nutch Seed Url Structure

2 properties

JSON STRUCTURE

Apache Nutch Service Config Structure

3 properties

JSON STRUCTURE

Apache Nutch Service Info Structure

1 properties

JSON STRUCTURE

Apache Nutch State Structure

0 properties

JSON STRUCTURE

Scroll within the panel for all 16 ·

Examples 16

Real request and response payloads are what turn a spec from abstract into obvious — and they're one of the twelve things an agent needs to call an API correctly on the first try.

Example request and response payloads for these APIs.

Scroll within the panel for all 16 ·

Security Posture 3

Authentication, domain security, vulnerability disclosure, and trust-center signals — the evidence that a provider takes security seriously enough to document it. We profile it because you can't govern what you can't see.

Authentication, domain security, vulnerability disclosure, and trust-center signals.

Apache Nutch Authentication

http · 1 scheme

SECURITY

Apache Nutch Domain Security

TLSv1.3 · HSTS · DMARC

SECURITY

Apache Nutch Vulnerability Disclosure

security.txt · contact published

SECURITY

Agentic Access 1

An x-agentic-access contract marks which operations are safe for an agent to run on its own and which need a human in the loop. It is the difference between an API an agent can use and one it can use safely.

Recommended x-agentic-access execution contracts for AI agents.

Apache Nutch Agentic Access

24 operations · 10 acting

24 operations · 10 acting

AGENTIC

Use Cases 6

What developers actually build with this provider — captured so the catalogue answers 'what is this for', not just 'what does this expose'.

What developers build with this provider.

Enterprise Search

Build enterprise search engines over internal or external web content using Nutch as the crawler and Solr/Elasticsearch as the search backend.

Research Data Collection

Academic and research teams use Nutch for large-scale systematic web data collection and indexing.

Intranet Document Search

Crawl and index intranet sites, wikis, and document repositories for internal enterprise search.

Web Archive Creation

Create structured web archives compatible with CommonCrawl format for long-term data preservation.

SEO and Content Monitoring

Monitor web content changes, track competitor sites, and analyze web structure at scale.

Custom Data Extraction Pipelines

Build custom extraction pipelines using Nutch plugin architecture for targeted data acquisition tasks.

Integrations 5

Pre-built integrations with other platforms tell you where this provider already fits in a stack.

Pre-built integrations with other platforms and tools.

Apache Solr

Native index writer plugin for indexing crawled content into Apache Solr for full-text search.

Elasticsearch

Index writer plugin for sending crawled content to Elasticsearch clusters.

Apache Hadoop

Core dependency providing distributed storage and processing via HDFS and MapReduce.

Apache Tika

Used for content detection and extraction from a wide range of document formats during parsing.

SolrCloud

Support for SolrCloud distributed search clusters for scalable indexing.

Resources

Every other property we hold for Apache Nutch — documentation, portals, status pages, policies, and corporate surface — grouped by the job it does, following the integrator's arc from getting started to running in production.

Get Started 1

Portal, sign-up, and the first successful call

Documentation 1

Reference material describing how the API behaves

Agent Surfaces 1

MCP servers, agent skills, and machine-readable catalogs

Learn 1

Tutorials, courses, talks, and written guidance

Commercial 1

Pricing, plans, and the legal terms of use

Company 1

The organization behind the API

← All providers · Data indexed from github.com/api-evangelist/apache-nutch · machine-readable index on apis.io