Important limitation
The script tag approach only fires when JavaScript executes in the browser. AI crawlers that do not run JavaScript — including many versions of GPTBot, ClaudeBot, and PerplexityBot — will not be tracked. Server-side middleware (Next.js, Express, Cloudflare Workers) is the recommended approach for reliable coverage.
Use this approach when you cannot modify your server configuration or when you need a quick setup as a temporary measure.
Middleware Setup
No package installation required. Add the following to the <head> of every page you want to track:
<!-- CrawlReady tracking pixel -->
<noscript>
<img
src="https://www.crawlready.app/api/v1/t/cr_live_xxxxx"
width="1"
height="1"
alt=""
style="display:none"
/>
</noscript>
Replace cr_live_xxxxx with your site key from the CrawlReady dashboard.
The <noscript> tag ensures this only fires in environments that do not execute JavaScript. When an AI crawler that supports image loading (but not JS) fetches the page, the 1×1 transparent GIF request is logged as a crawler visit with source pixel.
How it works
GET https://www.crawlready.app/api/v1/t/{siteKey} returns a 1×1 transparent GIF. The server reads the User-Agent header from the request and logs it if it matches a known AI crawler. The request is rate-limited per site key and deduplicated within a 1-second window, matching the same rules as the middleware ingest endpoint.
For sites that can run JavaScript
If your site runs JavaScript, you can also fire the beacon programmatically:
// Fire analytics beacon on page load (JavaScript-capable crawlers only)
(function () {
fetch('https://www.crawlready.app/api/v1/ingest', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
s: 'cr_live_xxxxx',
p: window.location.pathname,
b: navigator.userAgent,
t: Date.now(),
src: 'js',
}),
keepalive: true,
}).catch(function () {});
})();
Replace cr_live_xxxxx with your site key. This approach tracks crawlers that execute JavaScript, but most AI crawlers at the time of writing do not execute JavaScript on crawl.
Verify Your Integration
After deploying, visit one of your tracked pages with a bot User-Agent to trigger a pixel load, then check your CrawlReady dashboard for the visit.
You can also use the Verify Integration button in the dashboard — it sends a synthetic GPTBot request to your site. Note that the verification check requires your server to be publicly reachable and that the pixel or ingest call is made on the pages the checker visits.
Going Further
- Next.js integration — server-side tracking for full coverage
- Express integration — server-side tracking for Node.js apps
- Analytics Dashboard — understand your crawler traffic data
- Analytics Overview — what the metrics mean