ambolt
Menu

All endpoints

Sitemap and Robots.txt Doctor: Audit Crawl Files

Audit a website's robots.txt and XML sitemaps in one call: user-agent groups and blocked paths, sitemap declarations, whether each sitemap loads, URL counts (indexes followed to 10 files), lastmod coverage and age, duplicates, off-host and non-HTTPS URLs, size-limit breaches, and a sample of listed URLs checked for status. Returns a list of concrete issues.

x402: $0.005 per callMCP tool: sitemap_robots_doctor

Input

ParameterTypeDefaultDescription
site *stringDomain or URL of the site, for example example.com or https://example.com/blog.
sampleUrlsinteger5How many listed URLs to fetch and check for status (0 to skip).

* required

Source and freshness

Source and licence
Stated in every response (field source).
Last verified
2026-10-03 (hourly check against the live source)

Use it

HTTP (x402)

curl -i "https://api.ambolt.dev/v1/sitemap-robots-doctor?site=https%3A%2F%2Fwww.sitemaps.org&sampleUrls=3"

Returns 402 Payment Required with the price until an x402 payment is attached; @x402/fetch does this for you. See Get started.

MCP

{ "mcpServers": { "ambolt": { "url": "https://api.ambolt.dev/mcp" } } }
# then call the tool: sitemap_robots_doctor

Apify

Not published as an Actor yet.

Example response

{
  "site": "https://www.sitemaps.org",
  "robots": {
    "found": true,
    "bytes": 45,
    "groups": 0,
    "userAgents": [],
    "disallowRules": 0,
    "allowRules": 0,
    "crawlDelay": null,
    "sitemapDirectives": [
      "https://www.sitemaps.org/sitemap.xml"
    ],
    "blocksEverything": false
  },
  "sitemaps": [
    {
      "url": "https://www.sitemaps.org/sitemap.xml",
      "ok": true,
      "type": "urlset",
      "urls": 84,
      "bytes": 8622,
      "truncatedDownload": false
    }
  ],
  "stats": {
    "urlsRead": 84,
    "withLastmod": 84,
    "duplicates": 0,
    "offHost": 0,
    "nonHttps": 0,
    "newestLastmod": "2022-12-15",
    "oldestLastmod": "2016-11-21"
  },
  "sampleChecks": [
    {
      "url": "https://www.sitemaps.org/",
      "status": 200,
      "redirectedTo": null
    },
    {
      "url": "https://www.sitemaps.org/it/",
      "status": 200,
      "redirectedTo": null
    },
    {
      "url": "https://www.sitemaps.org/ro/",
      "status": 200,
      "redirectedTo": null
    }
  ],
  "issues": [],
  "summary": {
    "errors": 0,
    "warnings": 0
  },
  "checkedAt": "2026-10-02T23:02:47.122Z"
}

Good to know

Guides