Track AI crawlability over time

The problem

Whether an AI crawler gets a page is decided in more places than robots.txt. The content delivery network (CDN), the web application firewall and edge rules also answer the request, and they never read that file. A change to any of them, or to the hosting, can start turning crawlers away while robots.txt still says yes, and without a reading from before the change, nobody can show what moved.

When to use this playbook

You or your clients need to know whether AI crawlers can reach a site: before and after a change to the CDN, firewall or hosting, and after fixes, to show they held.

The workflow

1. 01

   Start from the Diagnostic

   The Discoverability pillar reads whether the site’s robots.txt lets AI retrieval bots in and whether the server turns away crawlers the rules invite. A Crawlability Scan measures the domain of a completed Diagnostic, so the Diagnostic comes first.

2. 02

   Take the first reading

   Run a Crawlability Scan. Importing the sitemap and choosing the pages is free; you pay for the crawl. Every approved page is checked against what robots.txt declares for each named AI crawler, and a sample of pages is requested as the crawlers themselves, next to a browser, to see what a request sent as each tested crawler was served. Run it before the change, or as soon as the question comes up.

3. 03

   Make the change or the fix

   The CDN rules, firewall settings, hosting move or robots.txt edit are the work of your team or the client’s developers. The scan says what a request sent as each tested crawler was served, not why, so the cause is read in those systems.

4. 04

   Re-scan the same pages

   Within 90 days of the first scan’s delivery, order a re-scan from the delivered scan in your dashboard, for less than the first scan cost. It fetches the pages approved the first time, reads the sitemap again, and repeats the crawler-versus-browser check on the same sampled pages. A re-scan of a re-scan counts from the same first delivery.

5. 05

   Compare the runs

   Crawlability Scan Trends lists every run of those pages and compares the latest with the one before it, page by page: pages that were gone and answer now, pages now gone (404 or 410), pages answering differently, pages taken out of the sitemap, and sampled pages whose crawler-versus-browser result changed.

6. 06

   Report it to the client

   When the Diagnostic a scan ran from is filed under the client’s workspace, the scan is filed there too. Your agency’s copy of that workspace’s progress report lists each scan delivered in the period and compares a re-scan with the run before it.

7. 07

   Share the scan itself

   To show the client a scan’s results, share it from the scan with their email address: they sign in with that address to read it, and you can revoke it at any time. A scan can also be printed or saved as a PDF.

What you use

• Diagnostic Ten pillars scored from public evidence in 5–10 minutes, with every finding turned into an action item.
• Crawlability Scan URL by URL, which pages AI agents and crawlers can reach and which they are turned away from.
• Crawlability re-scan The same approved pages fetched again within 90 days of the first scan’s delivery, for less than the first scan cost.
• Crawlability Scan Trends Every run of the same pages, with the latest compared with the one before it, page by page.
• Progress report One client over any period: what was wrong, what was done, what changed, what is still open.

What you end up with

A dated reading of what robots.txt declares for each page and what requests sent as the tested AI crawlers were served, taken before the change, the same pages read again after it, and a page-by-page comparison of the two.

The opportunity

Hosting, CDN and firewall work with a check on either side of it, and fixes you can show held, rather than asking the client to take your word for it. Each scan is a one-off order, bought for the change it checks.

What these scans do not do

• —Each scan is a reading at one point in time. Nothing runs on a schedule and nothing alerts you between scans; you decide when to look again.
• —A re-scan can be ordered for 90 days after the first scan is delivered, and re-scans do not extend that. A scan ordered afterwards starts a new series, compared only with its own re-scans.
• —A re-scan fetches the pages approved the first time. It does not look for new pages the sitemap leaves out, so the two runs match row for row.
• —If the first crawler-versus-browser sample cannot be read back, the re-scan draws a fresh one, and the comparison says it may cover different pages.
• —The crawler-versus-browser check sends the user-agent strings of the AI crawlers it tests from Yellowhead Digital’s servers. A CDN or firewall that verifies crawlers by their network address may answer the vendor’s own crawler differently. The check also covers a sample of pages for each template; every other page is read for what robots.txt declares.
• —Access to a client workspace does not show its scan results. In the product, a client sees a scan only when you share it with their email address, and the client’s copy of the progress report leaves scan results out.

What comes next