ambolt
Menu

Site owners

Ambolt Link Checker: information for site owners

How to recognise our checker, what it does to your site, and how to make it stop.

The Ambolt Link Checker finds broken links for website owners. To do that it has to request web addresses, including addresses on your site. This page explains how it behaves, how to recognise it and how to make it stop.

How it identifies itself

Every request carries this user agent:

AmboltLinkChecker/1.0 (+https://ambolt.dev/link-checker)

The URL in the user agent leads to the product page. If you see requests that use this name from somewhere else, they are not ours; tell us. The Link Checker docs explain how to allow the checker through a firewall or bot protection.

What it requests

  • Links that someone published. When a customer, the WordPress plugin or a visitor of the free check asks us to check a page, we request each link on that page once to see whether it answers. Your site gets a request only because a page somewhere links to it. These are single status requests, not a crawl.
  • Your own pages, only if you asked. We crawl a site's own pages only after its owner has proven control of the host (a DNS record, a meta tag, a file under /.well-known/ or the connected WordPress plugin). We never crawl an unverified site repeatedly, and the proof has to be renewed.
  • The free online check looks at one page the visitor entered and up to 60 links on it, once per visitor a day. At most 20 such checks a day can target the same host.

How politely

  • At most one request per second to the same host.
  • When it crawls the pages of a verified site, it reads that site's robots.txt and obeys the rules for the product token AmboltLinkChecker (or for * if there is none).
  • Requests to links on other people's sites are status requests only. They do not apply robots.txt because they are not a crawl, but they do honour our do-not-crawl list described below.
  • It does not log in, submit forms, run scripts or send cookies, and it does not keep the content of your pages: only the address, the status and a few facts about the answer.
  • It refuses private and internal network addresses, and it does not follow redirects blindly: each hop is checked again.

How to make it stop

Tell us the host that should not be requested. We put it, with its subdomains, on a do-not-crawl list, normally at once, and the checker stops asking for it for every customer and for the free check.

  • When the web app opens, use the form at app.ambolt.dev/lc/abuse. No account is needed. You get a confirmation by e-mail.
  • Until then, write to hello@ambolt.dev with the host (for example blog.example.com covers that host and its subdomains; example.com covers all of them).

If one of our customers has proven control of the host, they are the owner of the site as far as we can tell, so we review the request before acting. We read every report. The e-mail address you give us is used only to answer you and is removed 90 days after the report is handled.

You can also block the user agent in your firewall, with no need to ask us. Blocking is fine; it may show up as a "blocked" status for the customer who linked to you.

More