Skip to main content
AutoIndexNow

Docs

AutoIndexNowBot

AutoIndexNow only fetches from sites that a customer added and verified with a key file. This page is for site operators who see it in their logs.

User agents

  • AutoIndexNowBot/1.0 (+https://autoindexnow.com/docs/crawler) for robots.txt, sitemaps, key files and the lightweight page probe.
  • Mozilla/5.0 (compatible; AutoIndexNowBot/1.0; +https://autoindexnow.com/docs/crawler) when a page is rendered in a headless browser (a self-hosted Crawl4AI service) for change detection. Images, media and fonts are not loaded.

What it fetches and how often

Per verified site: robots.txt, the sitemaps it lists, and the pages in those sitemaps, at the customer's chosen interval (no more often than every 6 hours, depending on plan). Every check fetches each page's raw HTML. Server-rendered pages whose content in the raw HTML has not changed are not rendered again until their daily full render; pages that build their content with JavaScript are rendered on every check, within the account's daily render budget. Fetches follow at most five redirects, time out after 15 seconds, read at most 2 MB per page and 10 MB per sitemap, and never follow links on pages.

robots.txt

The crawler honours Disallow rules for AutoIndexNowBot or *. Disallowed URLs are neither rendered nor submitted to IndexNow. To block it entirely:

User-agent: AutoIndexNowBot
Disallow: /

When a URL is submitted

  • The first crawl of a site is a silent baseline: IndexNow is for changes from now on, not for pages that existed before.
  • A page added to the sitemap later, or a page whose title, description, main content, links or structured data changed, is submitted. A suspected edit is rendered twice before it counts, so render timing never does.
  • A page that starts returning 404 or 410, redirects, gets noindex or points its canonical elsewhere is submitted once, when consecutive checks have seen it for at least 12 hours, so a deploy or a short outage sends nothing. A page that leaves the sitemap keeps being checked for at least 7 days so a deletion is still noticed.
  • When many URLs appear at once (a new sitemap, a plan upgrade), they become a new baseline instead of being sent; you can submit them yourself from the URL list.
  • Unchanged pages, timestamps, counters, greetings, tokens and other dynamic noise are never submitted, and a URL accepted in the last 10 minutes is not sent again before that.

What it stores

Only a hash of each page's normalized content and operational metadata (status codes, timestamps). No HTML or rendered text is kept.

Credits

This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai).