A three-month research pilot between Cloudflare and OpenAI has produced its first confirmed result: by routing live network-observation signals from Cloudflare’s edge infrastructure into OpenAI’s web crawling pipeline, the two companies identified and indexed hundreds of millions of web pages that standard link-following crawlers would never have found.

Published today, the Cloudflare blog post today marks the first time a content delivery network has published quantified results showing how CDN-layer traffic signals close the structural gap in AI search indexing — and they set up a second, harder question: whether being indexed actually sends any traffic back to the websites that produced the content.

Pilot Found Hundreds of Millions of Pages Standard Crawlers Miss

The pilot, announced on July 8, 2026, used a deliberately bounded test cohort: approximately 1% of Cloudflare’s total network. Even within that constrained slice, the scale of what turned up exceeded what a link-following crawler would have reached. Cloudflare’s blog post states directly that the pilot “enabled OpenAI to discover and index hundreds of millions of new web pages.”

The gap the pilot was designed to close is structural, not incidental. Web crawlers discover content primarily by following links from one page to another. A page that few other websites point to may never get found, regardless of how accurate, current, or genuinely useful its contents are. Product manuals for specific appliance models, local business menus, government forms with current fees and deadlines, first-hand consumer reviews, niche how-to guides — these pages can have exactly what a user needs and still be invisible to a crawler that is navigating by link popularity.

As the Cloudflare blog describes link-invisible pages, someone fixing a washing machine needs the manual for their exact model, and someone applying for a passport needs the current requirements and fees. These pages can be valuable even if few other sites link to them.

How CDN Signals Close the Gap: From Traffic Observation to Crawl Queue

The mechanism behind the discovery gain is distinct from earlier approaches to the freshness problem — and it is worth understanding precisely what makes it work.

Traditional web indexing relies on two inputs: the link graph (which pages link to which other pages) and scheduled crawl revisits (how often a crawler returns to re-check a known URL). IndexNow and Crawler Hints — Cloudflare has processed more than 600 billion signals since 2021 — let site owners push a URL notification to search engines when content changes — but only if the site owner knows to send it. A link-invisible page whose owner is unaware of the problem never sends a notification.

CDN observation works differently: Cloudflare’s edge nodes sit in front of the HTTP requests made to websites it proxies, which means Cloudflare sees real user traffic flowing to those pages regardless of whether the pages are in any search index. When a page’s content changes, this produces observable cache-invalidation and cache-miss patterns at the edge. The pilot converted those observations into signals — content freshness indicators, traffic quality scores, and page-change timestamps — and fed them directly into OpenAI’s crawl scheduling pipeline.

The efficiency gains extended beyond discovery. Automated bot traffic regularly generates requests for pages that do not exist. Many sites redirect almost any address to their homepage. Others serve nearly identical content across dozens of URL variations. Cloudflare’s signals allowed OpenAI to identify these patterns before dispatching a crawl request at all, saving bandwidth and compute while concentrating crawler resources on pages likely to contain genuinely new content.

There is a key distinction between two separate improvements: the freshness problem (re-crawling known pages faster when they change) and the discovery problem (finding pages that were never in any index at all). CDN observation addresses both, because it operates from observed user traffic rather than from the existing link graph. A page invisible to link-following crawlers is not invisible to a CDN that is already serving its content to real users.

Safety Signals: One Compromised Page Without Penalizing the Domain

The pilot tested a second category of signals beyond freshness and discovery: safety filtering. Cloudflare’s network-level view can detect traffic anomalies consistent with page compromise — the pattern produced when an attacker injects a fake login form or malicious redirect into a specific page on an otherwise legitimate site.

Early pilot results suggest these signals can flag a single compromised URL before a crawler visits it, rather than treating the entire domain as suspect. For publishers, the distinction matters. A blanket domain penalty for a single compromised page can devastate a site’s search visibility; flagging only the affected URL limits collateral damage while still protecting users who might follow a ChatGPT citation to that page. As the Cloudflare blog notes on safety signals, a legitimate site can have a single compromised page — flagging that one page, rather than the whole domain, keeps people safe without penalizing the site owner.

What the Arrangement Shares — and What It Does Not

Cloudflare’s blog post is specific about the scope of the data exchange. The pilot uses “metadata about the web for search and discovery, rather than sharing page content for model training.” What flows from Cloudflare to OpenAI is observational — signals about when pages change, what traffic patterns look like, which URLs appear to have problems — not the page text itself.

Control over participation remains with site owners. Cloudflare’s AI Crawl Control dashboard allows operators to view crawler activity across their properties and set allow or block policies per individual crawler, per category, or per URL pattern. As the Cloudflare blog clarifies, a signal about a page does not grant permission to access it. A site that opts out via AI Crawl Control does not contribute signals, and a crawler still needs to fetch the page to use its content.

OpenAI VP of Research Nick Ryder, quoted at the July 2026 pilot announcement, stated the goal from OpenAI’s side: “Up-to-date information is important for delivering accurate answers to people using ChatGPT. Piloting this with Cloudflare will allow us to explore whether network-level insights can help us discover content more efficiently.”

Why Cloudflare Sells Tools to Block Crawlers and, Separately, to Help Them

The pilot sits alongside a set of products that appear to point in the opposite direction, and Cloudflare CEO Matthew Prince has addressed that apparent contradiction directly.

Over the past year, Cloudflare expanded its bot controls to distinguish three categories of AI crawlers: Search (indexing for retrieval), Agent (automated task execution), and Training (content for model development). It introduced a Pay Per Crawl monetization feature enabling per-request machine-to-machine payments from crawlers to content owners. In September 2026, it changed default settings for new domains so that Training and Agent crawlers are blocked by default on pages that display advertising. It also built “AI Labyrinth,” a feature that can feed synthetic content to crawlers that disregard site policies. All of this is covered in prior TechTimes Cloudflare crawler coverage.

The OpenAI pilot occupies a different lane: it helps a crawler that has already been permitted access find more of the web within the sites that have not blocked it. Cloudflare’s position is that AI Crawl Control gives site owners the choice to participate, making the two directions structurally consistent — one governs access, the other improves what happens inside permitted access.

What the arrangement also reveals is something the company’s prior product announcements did not foreground: a CDN with infrastructure in front of more than 20% of the global web occupies a passive observation position that functions as a structural input to AI search indexing. This is qualitatively different from prior forms of search influence, which required a crawler to request content. Cloudflare’s network-observation advantage operates regardless of whether any crawl partnership exists — which is why its decision to share those observations with one AI company, rather than all equally, carries competitive significance.

Prince has signaled awareness of that asymmetry in his engagement with UK regulators. He made the case to the CMA — the UK Competition and Markets Authority — that Google’s practice of bundling its search and AI crawlers into a single bot (Googlebot) unfairly advantages Google over competing AI search companies and should either be separated or subject to compensation requirements for content creators. The position is consistent with Cloudflare’s commercial interest in AI companies using its signal-sharing infrastructure — but it also reflects a genuine regulatory question about whether network-layer data advantages for specific AI companies require any non-discrimination framework.

Does AI Discovery Actually Send Traffic Back to Publishers?

The measurement that will determine this pilot’s real-world value for content creators has not yet been completed. Cloudflare’s blog describes the next phase as one that will measure post-discovery business outcomes for the websites whose pages were indexed.

That question lands in a context where publishers are already watching AI search’s impact on their referral traffic with considerable concern. Chartbeat data shared with Axios in March 2026 showed that small publishers — those receiving 1,000 to 10,000 daily page views — had lost 60% of their search referral traffic over the preceding two years. Medium-sized publishers lost 47%; large publishers lost 22%. ChatGPT referrals grew more than 200% over that period, but they still accounted for under 1% of all publisher page-view referrals.

This is the context in which “hundreds of millions of newly indexed pages” either matters to publishers or does not. Better indexing means ChatGPT can answer more questions using those pages — but if those answers are delivered within the chat interface without routing users back to the source, the publishers whose content enabled the answer receive no traffic from having been discovered. The Cloudflare blog states explicitly that measuring the downstream referral link is the next phase’s objective. Whether the results of that measurement will be published with the same specificity as the discovery numbers remains to be seen.

Frequently Asked Questions

How does ChatGPT Search find web pages that don’t have many links pointing to them?

Traditional web crawlers navigate by following links from one page to another — pages with few inbound links are often never reached, regardless of their content quality. The Cloudflare-OpenAI pilot substitutes a different signal: because Cloudflare’s infrastructure handles requests for more than 20% of the global web, its edge nodes can observe when real users access specific pages, when content changes, and which traffic patterns look healthy versus bot-generated. These observations are converted into signals that tell OpenAI’s crawler where to look — finding pages that exist and receive traffic even though no popular sites link to them.

Will being indexed by ChatGPT Search actually send traffic back to my website?

The first phase of the pilot answered the discovery question: hundreds of millions of link-invisible pages were found. The second question — whether that indexing translates into referrals — is explicitly what Cloudflare says the next phase will measure. As of today’s blog post, the traffic outcome is not yet confirmed. Context: Chartbeat data from March 2026 showed ChatGPT referrals represented less than 1% of publisher page-view traffic despite growing more than 200% year over year, suggesting that better AI indexing has not yet reliably converted into meaningful referral traffic at scale.

What is Cloudflare’s AI Crawl Control dashboard, and does it affect whether my site participates in this?

AI Crawl Control is a Cloudflare product that lets site owners set allow or block policies for individual crawlers, crawler categories (Search, Agent, Training), or specific URL patterns. Site owners who have not modified these settings participate in Cloudflare’s signal-sharing arrangement by default, but the arrangement shares only metadata — signals about page freshness, traffic patterns, and safety flags — not the page content itself. A site that wants to opt out can do so through the dashboard. Importantly, a discovery signal does not grant a crawler permission to access the page; that remains governed by robots.txt and access controls separately.

Does Cloudflare sharing network signals with OpenAI mean my page content is being used for AI training?

No. Cloudflare’s blog post on the pilot results specifies that the data flowing to OpenAI is “metadata about the web for search and discovery, rather than sharing page content for model training.” The signals describe how pages behave — when they change, what traffic looks like, whether any safety flags have been triggered — not what they contain. Access to the actual content requires a crawl request, which remains subject to normal robots.txt and AI Crawl Control settings.