Crawler

Ranklet-Audit

If Ranklet-Audit shows up in your logs, a Ranklet user asked us to check one page on your site for how likely AI search is to cite it. It reads that page, your robots.txt and your llms.txt, obeys robots.txt, and never follows links.

User agent

Ranklet-Audit/1.0 (+https://ranklet.io/bot)

Match on Ranklet-Audit; the version number will change.

When it visits

Only when someone using Ranklet runs a page audit on a specific URL of your site. It doesn't crawl on a schedule, discover pages, or come back on its own. Each visit is one audit of one page.

What it fetches

  1. /robots.txt and /llms.txt for your host.
  2. The page itself, as the raw HTML your server returns, but only if robots.txt allows Ranklet-Audit to read it.

If the page redirects to a different host, we follow the redirect, then read that host's robots.txt and discard the result if it disallows us.

robots.txt and llms.txt are cached for 24 hours per host, so auditing several pages on one site fetches them once.

What it doesn't do

  • Run JavaScript, or load images, stylesheets or scripts.
  • Follow links, submit forms, or sign in.
  • Fetch private or internal network addresses.

Limits

  • Follows up to 3 redirects.
  • Gives up on any request after 10 seconds.
  • Stops reading after 5 MB.

What happens to your page

We keep the audit's score and findings for the Ranklet user who ran it, not a copy of your HTML. To suggest a stronger opening, the page's first paragraph is sent to OpenAI's API and the suggestion is shown to that user. Ranklet doesn't train AI models.

How to block it

Add this to your robots.txt:

User-agent: Ranklet-Audit
Disallow: /

Use a narrower Disallow path to block only part of your site. Because robots.txt is cached, a change can take up to 24 hours to apply. The Ranklet user sees that your robots.txt disallows us, and no audit runs.

Contact

If Ranklet-Audit misbehaves on your site, email [email protected] with the URL and the time of the request, and we'll look into it.