The first sign was the crash. In late September 2025, multiple dashboards I monitor flatlined almost in unison. Clicks dove. Impressions collapsed. The same pattern appeared in third party panels tracking global platforms. It did not look like drift or seasonality. It looked synchronized and unusually fast.
At first I suspected AI Overviews alone. That would explain lower click through on affected queries. It would not explain a near simultaneous fall across unrelated domains and surfaces. The timing and the shape were too clean. The more likely story is a layered shift. A discovery choke at the search layer. A behavior shift at the results layer. And a new authentication wall at the network edge. Put together, that looks like the web building an immune response to automated agents.
- Google disabled the num=100 search parameter in September 2025. Bulk SERP retrieval fell from 100 results to 10 by default.
- Independent reporting shows lower click through when AI summaries appear compared with classic results pages.
- Automated traffic has been estimated near or above half of total web requests in recent years.
- CDNs introduced agent verification and throttling tools that can meter or block unverified automation at the edge.
Google disabled the num=100 parameter in early September. Rank trackers and scrapers lost easy access to full result sets. Discovery pipes shrank. Visibility models that relied on those responses shifted down even when page one ranking stayed stable.
When summaries appear, fewer people click out to publishers. That is measurable in independent studies. Even small exposure can move total click volume for certain query classes. It contains attention inside the results page.
Automated requests account for a large share of the web. Legitimate crawlers, SEO tools, price scrapers, and now AI agents. If discovery is throttled and agents crawl less, that synthetic layer drops out of aggregate metrics. Dashboards that blend human and automated activity will fall faster than real human demand.
CDN providers introduced agent verification and bot authentication. Sites can segment verified bots from unknown agents and apply allow, meter, or deny rules. That is a practical gate for machine traffic. Combined with search layer changes, it becomes a double lock. One at discovery. One at access.
Really Simple Licensing, or RSL, is an open standard that lets publishers declare machine readable licensing terms for AI and crawler use. Instead of a simple allow or disallow in robots.txt, RSL can express terms like attribution required, pay per crawl, subscription, or pay per inference. RSL is voluntary signaling by itself. The enforcement power arrives when RSL integrates with infrastructure. A CDN can check whether a requesting agent honors the license and then serve, meter, or block. In a tokenized future, this becomes the economic gate that sits beside the technical gates described above.
| Component | Observed change | Why dashboards can fall fast | Evidence vs theory |
|---|---|---|---|
| SERP data pipe num=100 removal | Fewer rows per fetch. Rank trackers undercount long tail visibility. | Visibility indices decline even if ranking is stable on page one or two. | Event confirmed. The throttle interpretation is my theory. |
| UX layer AI Overviews | Lower click through when summaries appear. Rare link clicks inside summaries. | Even small exposure moves total click volume for summary eligible queries. | Click effect reported. Treating it as a soft moat is my theory. |
| Automation mix bots and agents | Bots represent a large share of requests. Post change, many bots discover less and crawl less. | Analytics that under filter bots show larger declines than human only segments. | Bot share verified. Amplification mechanism is my theory. |
| Edge controls signed agents and pay per crawl | CDNs can fingerprint and meter AI agents. Sites can charge or block. | Agent swarms face new friction, reducing synthetic traffic while preserving verified bots. | Features exist. Framing them as an agent firewall is my theory. |
- The free crawl era is ending. Expect tokenized or paid access for agents that need bulk retrieval.
- Separate human and bot time series in GA4 and server logs to avoid false alarms.
- Measure click through separately where AI summaries appear and where they do not.
- Adopt edge policies for signed agents, verified bots, and pay per crawl.
- Decompose traffic: build strict bot rules and a human only view. Track both week over week.
- CTR study by class: compare impression and CTR shifts on summary eligible queries against control queries.
- Crawl telemetry: monitor Googlebot and third party crawler volumes at the edge to confirm structural declines after mid September.
- Rank tracker sanity checks: manually sample live results to detect artifacts from the parameter change.


