YellowheadBot

How to recognise it

Requests that identify as YellowheadBot carry this user agent:

YellowheadBot/1.0 (+https://yellowhead.digital/bot)

Some requests deliberately carry a different user agent. Those that reach websites are listed below under "Requests that use another user agent". Calls to a few data providers' own APIs identify us in the form each provider asks for, or alongside our own API key, instead.

When it visits

• When someone runs a Yellowhead Diagnostic or scan on your site, or re-checks a fix one of them suggested. That can be you, an agency or consultant working with you, or anyone else: a Diagnostic can be run on any public website. Our own team also runs Diagnostics for research.
• When a customer has scheduled repeat Diagnostics of your site, which run monthly or quarterly.
• When a customer follows your business in Yellowhead. Its public news, careers, leadership and changelog pages and its RSS feed can then be checked between Diagnostics.
• When another business's Diagnostic reaches your site. It checks that the competitors it names exist, including competitors suggested by an AI model, which can occasionally name the wrong domain. Each check is a HEAD request, then a GET if the HEAD request fails or gets a server error. It also checks that a form's destination answers when that destination is on another site, checks a few links on that business's pages, and reads the business's pages on social networks, job boards and app stores.

The same user agent is also on requests that are not reads of a business's site, for example: the sign-in metadata an AI assistant publishes when it connects to Yellowhead (a URL the assistant gives us), the IP-range documents that crawler vendors publish, YouTube's public lookup for a video's channel, the public registry of AI connectors, and lookups at public data services such as certificate-transparency logs and news indexes.

YellowheadBot does not crawl the web looking for sites to add. Every site it reads is connected to a business someone asked about, in one of the ways described above, or is a service like those just described.

What it requests

• Your homepage, and a bounded set of conventional paths: robots.txt, sitemaps, llms.txt, files that public standards place under /.well-known/, and common pages such as privacy, terms, about, contact, pricing, careers and blog. It also reads a few pages linked from your homepage or listed in your sitemap.
• Checks on what your homepage points to: the destinations of its forms, up to a dozen of its links, policy pages it links to on other hosts, and documents named in your AI-connector files.
• Other hostnames on your domain. It looks up a fixed list of common names (such as shop, login or api) and the names published for your domain in public certificate-transparency logs. It then fetches up to 40 of the hosts that exist, so you may see it on hostnames like staging.. It also sends HEAD requests to about a dozen names commonly used for server-side tagging (such as sst. or metrics.).
• A TLS handshake with your site to read its certificate: the opening of a secure (HTTPS) connection, which shows the certificate before any page is requested. That connection sends no HTTP request, so it carries no user agent.
• Its own HTTP requests are GET and HEAD only. It does not log in, submit a form or complete a checkout. A browser render runs your page's own scripts, and those may send requests of their own from the render addresses, including POST beacons to your analytics or consent tools.
• A browser render of your homepage and some other pages, from one of the locations listed below. A Diagnostic usually picks the location by your business's country, and can move once to another listed location if the first does not answer; an Accessibility Scan renders from the Netherlands.
• One observation of your homepage, which keeps the page open for up to 45 seconds from when it starts loading, plus a few seconds to finish reading it, and records the network requests the page makes. During it the browser scrolls the page and moves the pointer once. It never clicks, types, fills a form or answers a consent banner. For a business in the United States a second observation runs from the same address with the Global Privacy Control signal on, to see whether the site honours it. For a business in Canada outside Quebec, a second observation runs from Montréal.
• A Crawlability Scan or Accessibility Scan reads the pages its customer chose from your sitemap. Either scan makes its page reads and browser renders no faster than one a second. A Crawlability Scan also reads robots.txt and llms.txt, reads pages those pages link to that your sitemap does not list, and requests a sample of pages under AI crawlers' user agents (see below). It also checks up to 20 of the pages it read for a markdown version, with at most two requests per page: the same address asked for as markdown, then the address ending in .md. Where your llms.txt gives a .md link for the page, it requests that link instead. On a site that publishes no markdown versions, the .md requests come back as 404s. An Accessibility Scan renders each sampled page twice, at a mobile size and at a desktop size, and each render loads the page's own files at browser speed.
• Either scan stops for good when a page it reads answers 429, 503 or 999, or answers a direct read with a 403 that carries an anti-bot vendor's signature. A page that answers with another error status is recorded with that status, and the error page itself is never read as your content. An Accessibility Scan also sets aside, as not measured, a challenge page that its direct read gets with a normal status, and a render that answers 403, times out or fails to load, and moves on.
• A Diagnostic spaces most of its requests to your site about a second apart: the pages it reads, your robots.txt, sitemaps and llms.txt, and the other files and links it checks. A few smaller checks still run alongside those reads. When your site answers 429, 503 or 999, or a 403 that an anti-bot service marks as a block, it backs off and does not retry the refused request. A Diagnostic usually takes 5 to 10 minutes.

Requests that use another user agent

• Checking how your site treats AI crawlers. Some requests carry an AI crawler's published user agent, such as GPTBot or PerplexityBot, with a note appended that names Yellowhead Digital's UA-differential check (our user-agent differential check, which compares how your site answers different user agents) and links to our site, so your logs still name us. Requests with an ordinary desktop Chrome user agent, sent to the same pages, are the baseline the others are compared with. A site that serves these requests differently is what the check reports. They come from our servers, not from the AI vendors, and say nothing about what those vendors' own crawlers receive.
• App download links. To see where an app link sends an iPhone or a Mac, the link is followed with an iPhone or macOS Safari user agent.
• Instagram. Reading a business's public profile on instagram.com uses a desktop Chrome user agent.
• Outside services. Page speed is measured by Google's PageSpeed Insights, which fetches your site from Google's own infrastructure under Google's user agent. AI search services we query during a Diagnostic may also fetch your pages under their own user agents; we do not control that. None of these requests carry YellowheadBot or come from the addresses below.

robots.txt

YellowheadBot reads your robots.txt, because what it tells AI crawlers is one of the things a Diagnostic reports.

When someone runs a Diagnostic or scan of your site, YellowheadBot makes the bounded set of requests described above, once per run.

Three kinds of checks follow your robots.txt instead: scheduled repeat Diagnostics, checks on a business a customer follows, and re-checks of a fix. They obey the rules you set for YellowheadBot, or your rules for all crawlers if none name us. They don't request a page or file you disallow, and a scheduled Diagnostic doesn't run at all if you disallow the page it starts from. A missing robots.txt means no rules. If your server errors, rate-limits us or doesn't answer when we ask for robots.txt, that check treats every page as off limits.

Rules for all crawlers don't stop a Diagnostic or scan that someone runs, but rules that name YellowheadBot do. For anything else, see "If you would rather we didn't visit" below.

Where requests come from

Browser renders and observations come from these fixed addresses. Each location also makes a periodic geolocation lookup to a consent-platform provider from the same address:

The same list is published as JSON at https://yellowhead.digital/bot/ips.json.

Our other requests, the direct reads of pages and files and the AI-crawler check, come from our application servers, which do not have a fixed address. We are adding signed requests, so that they can be verified rather than taken on trust.

How to allowlist YellowheadBot

• Browser renders and observations: allow the user agent YellowheadBot/1.0 from the addresses above. Pairing the two is the stronger check, because anyone can copy a user agent.
• Direct reads of pages and files: allow requests whose user agent contains YellowheadBot/1.0. Today these can be recognised by the user agent alone, because our application servers have no fixed address; a rule that requires both the user agent and one of the addresses above will block them. A user agent can be copied by anyone, so a rule based on it alone also lets in anyone who pretends to be us. Signed requests will add a check that cannot be copied.

This does not cover the requests that use another user agent, listed above, or the outside services, which identify as themselves.

If you would rather we didn't visit

Blocking the YellowheadBot user agent, or the addresses above, stops the requests that carry them. It does not stop the requests that use another user agent, listed above, or the outside services, and our application servers have no fixed address to block. The complete route is to write to us at support@yellowhead.digital and ask us to stop reading your site.

We do not work around a block: no stealth browser automation and no captcha solving. A refusal status such as 403, 429 or 503 is recorded as a refusal and never scored as your page. Some page reads do not yet recognise a challenge page served with a normal status; the methodology page says which.