All guides

Sitemap and robots.txt audit API

Audit a website robots.txt and XML sitemaps in one call: blocked paths, sitemap health, URL counts, duplicates, off-host URLs and a ranked list of issues. Pay per audit.

Crawl problems often hide in two small files. One call reads a site's robots.txt and XML sitemaps and returns the user-agent groups and blocked paths, the sitemap declarations, whether each sitemap loads, URL counts (sitemap indexes are followed to ten files), lastmod coverage and age, duplicates, off-host and non-HTTPS URLs and size-limit breaches. A sample of listed URLs is checked for status.

The answer ends with a list of concrete issues, so a script or an agent can act on it without reading the raw files.

The calls

Each call returns 402 with the price until a payment is attached; an x402 client pays and retries automatically. Calls that fail are not charged. See Get started and the guide for bots and agents for code.

sitemap-robots-doctor $0.005 per call

Audit a website's robots.txt and XML sitemaps in one call: user-agent groups and blocked paths, sitemap declarations, whether each sitemap loads, URL counts (indexes followed to 10 files), lastmod coverage and age, duplicates, off-host and non-HTTPS URLs, size-limit breaches, and a sample of listed URLs checked for status.

curl -i "https://api.ambolt.dev/v1/sitemap-robots-doctor?site=https%3A%2F%2Fwww.sitemaps.org&sampleUrls=3"

{
  "site": "https://www.sitemaps.org",
  "robots": {
    "found": true,
    "bytes": 45,
    "groups": 0,
    "userAgents": [],
    "disallowRules": 0,
    "allowRules": 0,
    "crawlDelay": null,
    "sitemapDirectives": [
      "https://www.sitemaps.org/sitemap.xml"
    ],
    "blocksEverything": false
  },
  "sitemaps": [
    {
      "url": "https://www.sitemaps.org/sitemap.xml",
      "ok": true,
      "type": "urlset",
      "urls": 84,
      "bytes": 8622,
      "truncatedDownload": false
    }
  ],
  "stats": {
    "urlsRead": 84,
    "withLastmod": 84,
    "duplicates": 0,
    "offHost": 0,
    "nonHttps": 0,
    "newestLastmod": "2022-12-15",
    "oldestLastmod": "2016-11-21"
  },
  "sampleChecks": [
    {
      "url": "https://www.sitemaps.org/",
      "status": 200,
      "redirectedTo": null
    },
    {
      "url": "https://www.sitemaps.org/it/",
      "status": 200,
      "redirectedTo": null
    },
    {
      "url": "https://www.sitemaps.org/ro/",
      "status": 200,
      "redirectedTo": null
    }
  ],
  "issues": [],
  "summary": {
    "errors": 0,
    "warnings": 0
  },
  "checkedAt": "2026-10-02T23:02:47.122Z"
}

Questions

Does it crawl the whole site?

No. It reads robots.txt and the sitemaps and checks a small sample of the listed URLs.

How many sitemap files are followed?

Up to ten files from a sitemap index.

Is a failed audit charged?

No. Failed calls are not charged.

Use it from an AI agent

Every call is also an MCP tool: add https://api.ambolt.dev/mcp to your agent's server list.