Analytics tracks every AI crawler visit to your site — bot name, path, timestamp, and traffic source — and surfaces that data in the dashboard.
How data gets collected
Install the CrawlReady middleware in your Next.js app. On each inbound request, the middleware checks the User-Agent header against a registry of known AI crawlers. When it finds a match, it sends a beacon to the CrawlReady ingest API with:
- bot — crawler identifier (e.g.
GPTBot,ClaudeBot,Googlebot) - path — the URL path requested
- visited_at — request timestamp (UTC)
- source —
middlewareorscript
The beacon is fire-and-forget; it does not block the response to the crawler.
What the API stores
Each visit is stored as a row in crawler_visits. The dashboard queries this table to produce:
- Visit counts — total visits and unique pages over a selected period (1–90 days)
- Bot breakdown — visits grouped by crawler, with share percentage and trend vs. the previous period
- Top pages — the 10 most-crawled paths, ranked by visit count
- Timeseries — daily or hourly visit volume, broken down by bot category
Bot categories
Bots are classified into four categories: ai-training, ai-search, seo, and unknown. Verified bots are confirmed against known IP ranges or User-Agent strings. Unverified bots are classified by name pattern only.
See the Dashboard for a description of each panel and what the numbers mean.