Cloudflare installation

With your site behind Cloudflare, connect camolabs without changing normal browsing. The Worker passes traffic to your existing site by default and routes only eligible agent requests to camolabs.

1. A CNAME for the Agent domain.
2. A Worker on the main domain.

Before you start

You need:

• Access to the Cloudflare zone for yourdomain.com.
• Permission to add DNS records.
• Permission to create and deploy a Worker.
• A published page in camolabs.

Cloudflare docs:

• DNS records
• Worker routes

1. Save the Agent domain

In camolabs, go to Settings > Install.

Enter:

agents.yourdomain.com

Click Save.

2. Add the CNAME

In Cloudflare:

1. Open the yourdomain.com zone.
2. Go to DNS > Records.
3. Click Add record.
4. Add:

Field	Value
Type	CNAME
Name	agents
Target	app.camolabs.ai
Proxy status	DNS only

Keep this record DNS only. Cloudflare may warn that the record exposes an IP address because the target is app.camolabs.ai; that warning does not expose your site's origin server. Proxying the Agent domain can prevent camolabs from receiving the hostname it needs for the DNS test.

Save the record, then return to camolabs and click Test DNS.

3. Create the Worker

In Cloudflare:

1. Go to Workers & Pages.
2. Click Create.
3. Choose Start with Hello World or Create Worker.
4. Name it camolabs-agent-router.
5. Click Deploy.
6. Open the Worker and click Edit code.

Replace the starter code with this Worker template:

const ROUTER_VERSION = "2026-06-19";
const POLICY_PATH = "/.well-known/camolabs-routing-policy.json";
const DEFAULT_HTML_TIMEOUT_MS = 2500;
const DEFAULT_POLICY_TIMEOUT_MS = 2500;
const DEFAULT_AGENT_PATTERN =
  "\\bbot\\b|crawler|spider|headless|python-requests|python-httpx|aiohttp|curl/|wget/|axios/|node-fetch|undici|playwright|puppeteer|selenium|go-http-client|okhttp|scrapy|java/|libwww|http_request";
const DEFAULT_SEARCH_CRAWLER_PATTERN =
  "googlebot|bingbot|bingpreview|slurp|duckduckbot|baiduspider|yandexbot|applebot|facebookexternalhit|linkedinbot";
const DEFAULT_EXCLUDED_PATH_PATTERN =
  "^/(?:api(?:/|$)|_next(?:/|$)|static(?:/|$)|assets(?:/|$)|favicon\\.ico$|robots\\.txt$|sitemap\\.xml$|.*\\.(?:avif|css|gif|ico|jpg|jpeg|js|json|map|mjs|mp4|pdf|png|svg|txt|webm|webp|woff|woff2|xml)$)";
const BROWSER_SIGNAL_HEADERS = ["sec-fetch-mode", "sec-fetch-dest", "sec-ch-ua"];
const FORWARDED_REQUEST_HEADERS = [
  "accept",
  "accept-language",
  "referer",
  "sec-fetch-mode",
  "sec-fetch-dest",
  "sec-ch-ua",
  "user-agent",
];
const FORWARDED_EDGE_HEADERS = ["cf-bot-score", "cf-connecting-ip", "cf-ipcountry", "cf-ray", "cf-verified-bot"];

let policyCache = null;

function matches(pattern, value) {
  if (!pattern || !value) return false;
  try {
    return new RegExp(pattern, "i").test(value);
  } catch {
    return false;
  }
}

function hasBrowserSignal(request) {
  return BROWSER_SIGNAL_HEADERS.some((name) => request.headers.has(name));
}

function looksAutomated(request, policy) {
  return matches(
    policy?.automation_user_agent_pattern || DEFAULT_AGENT_PATTERN,
    request.headers.get("user-agent") || ""
  );
}

function cleanOriginRequest(request, policy) {
  const url = new URL(request.url);
  let changed = false;
  for (const name of policy?.validation_query_params || [
    "camolabs_validation_run_id",
    "camolabs_validation_attempt_id",
  ]) {
    if (url.searchParams.has(name)) {
      url.searchParams.delete(name);
      changed = true;
    }
  }
  const originParam = policy?.origin_request_query_param || "camolabs_origin";
  if (url.searchParams.has(originParam)) {
    url.searchParams.delete(originParam);
    changed = true;
  }
  return changed ? new Request(url.href, request) : request;
}

async function fetchWithTimeout(url, init, timeoutMs) {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), timeoutMs);
  try {
    return await fetch(url, { ...init, signal: controller.signal });
  } finally {
    clearTimeout(timeout);
  }
}

async function loadPolicy(agentOrigin) {
  const now = Date.now();
  if (policyCache && policyCache.expiresAt > now) return policyCache.value;
  try {
    const response = await fetchWithTimeout(
      new URL(POLICY_PATH, agentOrigin),
      { headers: { accept: "application/json" } },
      Math.max(Number(policyCache?.value?.policy_timeout_ms) || DEFAULT_POLICY_TIMEOUT_MS, DEFAULT_POLICY_TIMEOUT_MS)
    );
    if (!response.ok) return policyCache?.value || null;
    const policy = await response.json();
    const ttl = Math.max(30, Math.min(Number(policy.cache_ttl_seconds) || 300, 3600));
    policyCache = { value: policy, expiresAt: now + ttl * 1000 };
    return policy;
  } catch {
    return policyCache?.value || null;
  }
}

function shouldAskCamolabs(request, sourceUrl, policy) {
  if (!policy?.routing_enabled) return false;
  if (!Array.isArray(policy.routed_methods) || !policy.routed_methods.includes(request.method)) return false;
  if (sourceUrl.searchParams.has(policy.origin_request_query_param || "camolabs_origin")) return false;
  if (matches(policy.excluded_path_pattern || DEFAULT_EXCLUDED_PATH_PATTERN, sourceUrl.pathname)) return false;
  if (
    matches(
      policy.search_crawler_user_agent_pattern || DEFAULT_SEARCH_CRAWLER_PATTERN,
      request.headers.get("user-agent") || ""
    )
  )
    return false;
  if (hasBrowserSignal(request) && !looksAutomated(request, policy)) return false;
  return true;
}

function agentHeaders(request, sourceUrl, policy) {
  const headers = new Headers();
  for (const name of policy?.forwarded_request_headers || FORWARDED_REQUEST_HEADERS) {
    const value = request.headers.get(name);
    if (value) headers.set(name, value);
  }
  for (const name of policy?.forwarded_edge_headers || FORWARDED_EDGE_HEADERS) {
    const value = request.headers.get(name);
    if (value) headers.set(name, value);
  }
  headers.set("x-camolabs-original-url", sourceUrl.href);
  headers.set("x-camolabs-origin-host", sourceUrl.host);
  headers.set("x-camolabs-origin-proto", sourceUrl.protocol.replace(":", ""));
  headers.set("x-camolabs-router-version", ROUTER_VERSION);
  return headers;
}

function isAgentPage(response, policy) {
  return (
    response.status === 200 &&
    (response.headers.get("content-type") || "").includes("text/html") &&
    response.headers.get(policy?.agent_page_outcome_header || "x-camolabs-agent-page-outcome") ===
      (policy?.served_agent_page_outcome || "served_agent_page")
  );
}

export default {
  async fetch(request, env) {
    if (request.method !== "GET") return fetch(request);
    const agentOrigin = env.CAMOLABS_AGENT_ORIGIN;
    if (!agentOrigin) return fetch(request);

    const sourceUrl = new URL(request.url);
    if (hasBrowserSignal(request) && !looksAutomated(request, null)) return fetch(cleanOriginRequest(request, null));

    const policy = await loadPolicy(agentOrigin);
    if (!shouldAskCamolabs(request, sourceUrl, policy)) return fetch(cleanOriginRequest(request, policy));

    const agentUrl = new URL(`${sourceUrl.pathname}${sourceUrl.search}`, agentOrigin);
    try {
      const response = await fetchWithTimeout(
        agentUrl,
        { method: "GET", headers: agentHeaders(request, sourceUrl, policy) },
        Math.max(Number(policy?.html_timeout_ms) || DEFAULT_HTML_TIMEOUT_MS, DEFAULT_HTML_TIMEOUT_MS)
      );
      return isAgentPage(response, policy) ? response : fetch(cleanOriginRequest(request, policy));
    } catch {
      return fetch(cleanOriginRequest(request, policy));
    }
  },
};

Click Deploy.

Then add the Agent origin as a Text variable:

1. Open the Worker.
2. Go to Settings > Variables.
3. Add a Text variable:

Variable	Value
CAMOLABS_AGENT_ORIGIN	https://agents.yourdomain.com

Replace https://agents.yourdomain.com with the Agent domain you saved in camolabs.

If your main domain already uses a Worker on the same route, merge this logic into that Worker instead of creating a second Worker for the same path.

4. Route the Worker

Add a route for the public pages where eligible agent requests can receive Agent Pages. Normal browser traffic on the same route continues to your existing site.

In Cloudflare:

1. Open the yourdomain.com zone.
2. Go to Workers Routes.
3. Click Add route.
4. In Route, enter the hostname and path pattern to protect.
5. In Worker, select the Worker you created in step 3.
6. Click Save.

If your public site is served from the apex domain, use:

yourdomain.com/*

If your public site is served from www, use:

www.yourdomain.com/*

Use narrower routes when only part of the site should be eligible for Agent Pages:

www.yourdomain.com/pricing/*
www.yourdomain.com/docs/*

After saving, your route table should include one route from your public site to your camolabs router Worker:

Route	Worker
yourdomain.com/*	camolabs-agent-router
www.yourdomain.com/*	camolabs-agent-router

You only need the row that matches the hostname people use to visit your site.

Do not route the Agent domain through this Worker. The Agent domain should point directly to camolabs through the CNAME from step 2.

If your main domain already has a Worker route, Cloudflare will not run two Workers for the same route. Add the camolabs routing logic to the existing Worker instead.

5. Test routing

In camolabs, go to Settings > Install and click Test routing.

You can also run:

curl -i https://www.yourdomain.com/pricing \
  -H "User-Agent: python-requests/2.31"

Expected Agent Page response:

HTTP/2 200
x-camolabs-agent-page-outcome: served_agent_page
content-type: text/html; charset=utf-8

If the Agent Page is missing, unpublished, or camolabs is unavailable, the Worker continues to the existing site.

Runtime behavior

The Worker:

• Passes traffic to your existing site by default.
• Asks camolabs only when the routing policy allows the request.
• Skips browser navigations, search crawlers, assets, APIs, and excluded paths.
• Requests the same path from https://agents.yourdomain.com.
• Returns camolabs output only when the response is an Agent Page.
• Removes camolabs validation and origin request parameters before sending traffic to origin.