Track AI crawlers

Confirm that AI crawlers can reach your content, and see which ones visit and how often — with optional server snippets for richer data.

If an AI assistant can't crawl your pages, it can't cite you — no matter how good your content is. Crawler tracking shows you which AI crawlers are visiting your site and how often, so you can confirm the assistants that matter are actually reaching your content.

What you'll need

A connected, verified site.
Optional: access to your hosting or CDN configuration if you want to forward richer request logs (see Forward your server logs).

See who's crawling you

Open Crawler logs

In AI Visibility, open the Crawler logs view. It lists the AI crawlers seen visiting your site, how many requests each made, and when they last came by.

Check the crawlers that matter

Look for the assistants you care about being visible in. If a crawler you'd expect is absent, that's the first thing to fix — an assistant that never fetches your pages will never cite them.

Confirm access

Make sure your robots.txt and any firewall or bot rules aren't blocking the AI crawlers you want. Blocking them is the most common reason a site is invisible to an assistant.

Forward your server logs

Basic crawler detection works with no setup. To get the most complete picture, you can forward request logs from your own hosting so every crawler hit is captured. These snippets run on your infrastructure — they don't expose any secret, and you can remove them anytime.

Cloudflare

If your site is behind Cloudflare, add a Worker (or Log Push job) that forwards requests whose user-agent looks like an AI crawler. A minimal Worker looks like this:

export default {
  async fetch(request, env, ctx) {
    const ua = request.headers.get('user-agent') ?? '';
    // Forward only AI-crawler-looking requests to your log endpoint.
    if (/GPTBot|ClaudeBot|PerplexityBot|Google-Extended|Bytespider/i.test(ua)) {
      ctx.waitUntil(
        fetch(env.CRAWLER_LOG_ENDPOINT, {
          method: 'POST',
          headers: { 'content-type': 'application/json' },
          body: JSON.stringify({
            ua,
            path: new URL(request.url).pathname,
            ts: Date.now(),
          }),
        }),
      );
    }
    return fetch(request);
  },
};

Set CRAWLER_LOG_ENDPOINT as a Worker environment variable using the endpoint shown in the Crawler logs setup panel. Never hard-code the endpoint or any token into a public repo.

Vercel

On Vercel, add middleware that records AI-crawler requests and forwards them to the same endpoint:

// middleware.ts
import { NextResponse } from 'next/server';
import type { NextRequest } from 'next/server';

export function middleware(request: NextRequest) {
  const ua = request.headers.get('user-agent') ?? '';
  if (/GPTBot|ClaudeBot|PerplexityBot|Google-Extended|Bytespider/i.test(ua)) {
    // Fire-and-forget; don't block the response.
    fetch(process.env.CRAWLER_LOG_ENDPOINT!, {
      method: 'POST',
      headers: { 'content-type': 'application/json' },
      body: JSON.stringify({ ua, path: request.nextUrl.pathname, ts: Date.now() }),
    }).catch(() => {});
  }
  return NextResponse.next();
}

Add CRAWLER_LOG_ENDPOINT as a Vercel environment variable — keep it out of your source.

Keep any endpoint URL or token in an environment variable, never committed to your repository. These snippets only read request metadata (user-agent, path, time) — don't add request bodies, cookies, or headers that could contain personal data.

Plan availability

Crawler logs are part of AI Visibility on paid plans. Basic detection needs no setup; log forwarding is optional and free to add on your own hosting.

FAQs

Do I have to add the snippets? No. Crawler logs work without them. Forwarding your own logs just gives a more complete count of crawler visits.

An assistant I want isn't crawling me. What do I do? Check robots.txt and any bot-blocking rules at your CDN or firewall, and make sure the crawler's user-agent is allowed. Then give it time to return.

Is any secret exposed by these snippets? No — they read request metadata only, and the log endpoint belongs to your account. Keep the endpoint in an environment variable.

On this page