Best Buy: The Bot Wall Problem
We asked an AI agent to find USB-C SSDs on Best Buy. Three different tools failed. Their entire product catalog is invisible to AI shopping agents. We built the pipeline that breaks through.
reduction
time
agents to parse
extracted
The Problem
When an AI agent is asked "find me a good external SSD under $150," it doesn't Google it and click through ten blue links. It tries to fetch product data directly from retailer sites. Here's what happens when it tries Best Buy:
| Method | What Happened | Why |
|---|---|---|
| Direct HTTP fetch | Connection reset | Anti-bot middleware rejected the request |
| Headless browser (Chromium) | Timeout | Bot detection fingerprinting blocked headless instances |
| AgentCDN server-side fetch | Timeout after 10s | Indistinguishable from a scraper to the bot wall |
We ran our agent benchmark against bestbuy.com. The result: 0 out of 10. Grade F. Not because Best Buy has bad SEO—they have great SEO. But the bot wall that protects them from scrapers also protects them from their next generation of customers.
Every product in Best Buy's catalog is invisible to AI shopping agents. Every agent-driven product recommendation right now excludes them.
Why This Matters
This isn't a Best Buy problem. It's an industry problem. Every major e-commerce site with enterprise bot protection has it:
- Best Buy — Akamai Bot Manager
- Amazon — Custom anti-bot + CAPTCHA
- Walmart — PerimeterX
- Target — Akamai
- Home Depot — Imperva
These bot walls were built to stop scrapers and credential stuffers. They do that well. But they also block AI shopping agents—the same agents that Stripe is building commerce infrastructure for, that Coinbase is enabling with agentic wallets, and that are driving a growing share of online purchasing.
The retailer whose products agents can read wins the recommendation. The one behind a bot wall loses the sale.
The Solution
We built a two-stage pipeline that solves both problems: getting past the bot wall and making the data agent-readable.
Stage 1
Agent-Browser (Headed Mode)
A real, visible Chromium instance—not headless. Full browser profile with normal fingerprints, WebGL rendering, and behavioral signals. Indistinguishable from a human visitor. Best Buy's Akamai Bot Manager serves the full page. The browser extracts the raw DOM: 1,003 KB of HTML including scripts, styles, tracking pixels, navigation chrome, ad containers, cookie banners, and filter sidebars.
Stage 2
AgentCDN Cleaning Pipeline
The raw HTML feeds into AgentCDN's cleaning engine. Regex-based processing strips non-content elements: scripts, styles, tracking pixels, navigation chrome, ad containers, cookie banners. Extracts main content. Converts to structured markdown. Processing time: 25ms.
Output
Agent-Ready Product Data
Clean, structured product data the agent can immediately parse and act on. Product names, prices, sale prices, ratings, review counts—all in a format that costs agents pennies instead of dollars to process.
The Results
| Metric | Before (Raw) | After (Pipeline) | Change |
|---|---|---|---|
| HTML size | 1,003 KB | 9.8 KB | 99.0% smaller |
| Token count | ~250,000 tokens | ~2,000 tokens | 125x reduction |
| Agent processing cost | $0.75/page | $0.006/page | 125x cheaper |
| Cleaning time | N/A (blocked) | 25ms | Instant |
| Products extracted | 0 (blocked) | 18 products | Full page 1 |
| Agent success rate | 0% | 100% | Operational |
What the Agent Sees
After the pipeline, an AI agent gets clean, structured product data it can immediately use:
| Product | Price | Was | Rating | Reviews |
|---|---|---|---|---|
| Crucial X9 Pro 1TB | $104.99 | $119.99 | 4.7 | 383 |
| SanDisk Extreme 2TB | $319.99 | $459.99 | 4.8 | 2,437 |
| SanDisk Extreme Pro 2TB | $329.99 | $488.99 | 4.4 | 432 |
| Crucial X10 Pro 4TB | $414.99 | $524.99 | 4.3 | 167 |
| Crucial X9 Pro Mac 2TB | $189.99 | — | 4.8 | 158 |
5 of 18 products shown. No hallucinated prices. No outdated data. Real-time from the live storefront.
Why Each Tool Alone Fails
The pipeline works because it combines two capabilities that are useless in isolation:
| Capability | Agent-Browser Alone | AgentCDN Alone | Combined |
|---|---|---|---|
| Bypass bot detection | Yes (headed) | No | Yes |
| JavaScript rendering | Yes | No | Yes |
| Content cleaning | No | Yes | Yes |
| Structured markdown | No | Yes | Yes |
| Token efficiency | ~250K tokens | ~2K tokens | ~2K tokens |
| Cost per page | $0.75+ | $0.006 | $0.006 |
Agent-browser can get past the wall but returns 1MB of raw DOM—expensive and noisy. AgentCDN can clean HTML into agent-ready markdown but can't get past bot protection. Together, they solve both problems.
The Opportunity
Every major e-commerce retailer is sitting behind a bot wall right now. Their competitors are too. The first retailer to create an authenticated agent channel—a way for verified AI agents to access product data without triggering the bot wall—captures a channel that everyone else is locked out of.
As Stripe's Agent Commerce Suite, Coinbase's agentic wallets, and Google's Universal Commerce Protocol drive more purchasing through AI agents, the retailers whose products agents can read will get the recommendations. The ones behind bot walls will lose sales they never knew existed.
We build the bridge. Whether that's an AgentCDN integration layer, an authenticated agent API, or a full agentic SEO optimization—we make your products the ones agents recommend.
Is Your Site Behind a Bot Wall?
We'll run the same benchmark against your site and show you what agents see—or don't.
Get Your Free Audit