AI crawler licensing, how publishers detect AI bots and turn access into revenue

A new revenue stream with real uncertainty

For years, the deal between websites and crawlers was simple. Search engines crawled pages and sent visitors back. AI crawlers change that balance. They collect content to train models or to answer questions directly, and many answers send few visitors to the source.

Some publishers have responded by blocking AI crawlers. Others have signed licensing agreements. A growing set of tools lets sites charge per crawl. The market is young, terms vary a great deal and most amounts are not public.

Monetisation treats AI crawler licensing as one of 16 revenue streams. It is honest about what can and cannot be estimated, and it starts with the part you can measure precisely: what AI crawlers are already taking.

Step one, detect AI crawlers

In server and CDN logs

AI crawlers identify themselves with user agent strings that their operators publish, for example GPTBot, CCBot, PerplexityBot, Bytespider, Applebot-Extended and similar. Your server or CDN logs record every request with its user agent, the page requested and the size of the response.

From the logs you can count:

• Requests per crawler per day
• Pages and sections crawled most often
• Bytes transferred, which shows the cost of serving them
• Share of total requests that come from AI crawlers

User agents can be faked, so a careful count verifies requests against the IP ranges published by each operator. Analytics tools that run on JavaScript usually miss crawlers entirely, which is why many site owners underestimate the volume.

In robots.txt

Your robots.txt shows your current policy. Some AI companies use separate tokens for training and for live retrieval, such as Google-Extended for training use alongside the normal search crawler. Reading the file tells you which crawlers you allow, which you block, and whether the policy matches what you intend.

What Monetisation reads

Monetisation crawls your public robots.txt and pages, and accepts log exports as CSV. It then reports crawler activity by operator and section, flags crawlers that ignore your robots.txt rules, and compares the crawl volume with your human traffic from Google Analytics 4.

Step two, understand the licensing models

Machine-readable licensing terms, declared alongside robots.txt, are also emerging so that crawlers can find a site's terms automatically. Adoption is still uneven.

Step three, estimate with honest ranges

Here is where Monetisation is deliberately cautious. Licensing amounts for individual sites are rarely published, and the deals that are public involve large publishers. So the estimate is built from inputs you control and labelled with low confidence unless you have actual terms.

Take a hypothetical specialist site with 2 million AI crawler requests a month across 40,000 articles.

The spread is wide on purpose. The per-request prices are assumptions, not market rates, and the card says so. Confidence stays low until you enter real offers or signed terms, at which point the estimate uses them. This is an estimate, not a guarantee of results, and the general method is explained on how estimates work.

Weighing licensing against other streams

AI licensing does not stand alone. Blocking AI crawlers may reduce how often your site appears in AI answers, which can lower referral traffic that feeds ads, affiliate and memberships. Allowing them free may cost you nothing today and something later.

Monetisation puts the licensing estimate next to your other streams by revenue per visitor, so the trade-off is visible. If your comparison pages earn well from affiliate links, for example, losing AI referral traffic to them matters more than losing it to news pages. See affiliate monetization and the wider website monetization platform page for how streams are compared.

Practical steps for publishers

1. Measure first. Count AI crawler requests by operator and section for at least a month.
2. Write a deliberate robots.txt policy. Decide separately for training crawlers and retrieval crawlers.
3. Enforce where needed. Block crawlers that ignore robots.txt at the server or network edge.
4. Publish terms. State the licensing terms on your site so operators can find them.
5. Test one model. Try pay per crawl or approach a licensing intermediary, and measure what happens to crawl volume and referrals.

Monetisation records each policy change as a dated intervention and compares crawler volume, referral traffic and revenue against the baseline trend, ending with a verdict of win, flat, loss or not enough data.

Common mistakes

• Relying on analytics alone. JavaScript analytics rarely records crawlers, so the volume looks smaller than it is.
• Blocking everything at once. A blanket block can remove the site from AI answers that send valuable visitors, before any licensing alternative exists.
• Trusting user agents without checks. Some scrapers copy the names of well-known crawlers. Verify against published IP ranges.
• Treating a first offer as the market rate. With so few public terms, the first number offered is a starting point, not a benchmark.

Who this opportunity suits

AI crawler licensing matters most for sites with large archives of original content: news and specialist publishers, reference sites, forums, and companies holding unique datasets. Our page on content monetization for publishers covers the other streams those sites usually combine with it.

Where to start

Run a sample audit from the homepage and choose the publisher business type. The result includes a revenue stack map showing whether AI crawler licensing is relevant to your site. To analyse crawler logs and compare streams, get started on any plan. See pricing for what each plan includes.