Short answer: Websites block AI agents when automated access conflicts with their security rules, service terms, or commercial decisions. A user approving a task does not guarantee that every website will accept the agent. For a business, access belongs in the buying decision alongside capability and cost.
While researching this article on October 20, we tried to read the same public X post through two tools. Our text retrieval tool reported 403 Forbidden. A browser view opened the post and its replies.
That is one observation, not a benchmark. It does not establish why the first request failed, or prove that X deliberately blocked an AI agent. It does show why “the agent has web access” is too vague to be an acceptance test.
We could finish checking the source because we had another working route. A recurring business workflow needs to know what happens when it does not.
Businesses are being sold software that can work across the internet. The internet gets a vote.
In its October 9 reporting on agent access, TechTarget described companies deciding whether outside systems can act on their platforms. Our take: a better model can improve the work without removing the dependency on someone else letting it through the door.
The user says yes. The website says no.
The Amazon–Perplexity dispute made this conflict unusually explicit. In November 2025, Amazon publicly objected to Perplexity’s Comet shopping agent and said third-party purchasing applications should identify themselves and respect a service provider’s decision about participation. Amazon’s statement framed the issue around transparency and customer experience.
Perplexity argued that users should be able to choose an assistant to shop on their behalf. Its response, “Bullying Is Not Innovation,” cast the conflict as a fight over user choice and control of the shopping experience.
Those are the companies’ competing positions. They expose a dependency that extends well beyond shopping: a customer authorizing an assistant does not, by itself, guarantee that a platform will accept its automated activity.
A business can buy a capable agent, give it a legitimate task, and still encounter an access restriction. More intelligence does not automatically resolve that restriction.
“AI bot” hides three different jobs
Part of the confusion comes from treating every automated visitor as the same thing. Cloudflare’s current documentation separates three AI-related behaviors: search, training, and agents acting for a person.
| Activity | Purpose | Decision for a site owner |
|---|---|---|
| Search | Collect and index content for discovery | Which search services should find the content? |
| Training | Collect material to train or fine-tune models | Which training uses are acceptable? |
| Agent | Retrieve information or act for a user | Which user-directed actions should be allowed? |
A publisher might welcome discovery, reject training, and negotiate a separate arrangement for agent access. A retailer might want automated purchases through an approved integration while rejecting an unidentified browser agent. One system can also perform more than one kind of activity.
This distinction matters when reading the headlines. In a June 3 X post, Cloudflare CEO Matthew Prince discussed bots overtaking human traffic. In the replies, Curt Tigges pointed out the distinction between bots and agents. A broad measure of automated traffic cannot, on its own, tell us how much work personal AI assistants are completing.
Welp, that happened faster than I predicted.
I think it's worth distinguishing between "bots" and "agents"
Access is becoming something websites manage explicitly
Site owners already have controls beyond a polite request in a robots.txt file. Cloudflare’s AI Crawl Control lets them allow or block identified crawlers and inspect request activity. Its documentation also describes configurable block responses.
There is a commercial dimension, too. Cloudflare documents Pay Per Crawl, currently described there as a closed beta, for charging participating AI crawlers for content access. This is a crawler-access mechanism; it does not grant a browser agent universal permission to use a website or transact.
These controls make the direction clear: automated access can depend on identity, purpose, technical rules, and commercial arrangements. A public URL is not a promise that every automated client will receive the same access.
The failure you should worry about is an incomplete answer that looks finished
Our agent buying scorecard separates capability from operations. Access is another operational condition to test before relying on a demo.
A visible error is manageable. Someone sees the problem and takes over.
The harder failure is a completed-looking deliverable built from incomplete access. A supplier comparison omits a source. A market brief misses a publication. An agent finds a product description but cannot verify stock.
These are illustrative failure modes, not results from a OneWave benchmark. They show why an evaluation should inspect both the answer and the information the agent actually obtained.
An agent should distinguish “I checked and found nothing” from “I could not check.” It should also distinguish reading a record, drafting a change, and successfully saving that change. Those are different outcomes.
The most useful question in an agent demo may be: “Show me what you could not access.”
Before you buy an agent, ask for an access test
Start with a bounded task, such as one of these simple business automations. Give the vendor one representative workflow using the systems your team actually relies on. Include the awkward supplier portal, the account with restricted permissions, and the document that requires a current login.
Then ask for five things:
- An access map. Which steps use an authorized API or connector? Which rely on a browser? Which require a person? A connector still needs the right permissions and supported actions. Our MCP explainer covers the connection layer.
- A source record. What did the agent read, when did it read it, and which required sources remained inaccessible?
- A completion check. What evidence confirms that the requested action succeeded in the destination system?
- A failure handoff. When access breaks, does the agent pause, explain the missing step, and let someone resume without duplicating completed work?
- A maintenance owner. Who notices when a login, integration, website, or access policy changes—and who fixes it?
Test the exception path deliberately. If the agent cannot read a required source, the resulting report should make that gap visible. If it cannot complete an action, it should leave a clear status rather than a vague assurance.
We turned those checks into a downloadable access acceptance worksheet. Use one row per required system or source. Record the expected action, observed result, evidence, and person who owns the fallback. The worksheet is blank by design; it does not claim measured vendor results.
Build around access you can depend on
Our recommendation is to use supported, authorized integrations for recurring work where they cover the task, use browser interaction where permitted, and keep an explicit human handoff for steps the agent cannot complete.
As we explain in our cloud-versus-local cost analysis, the useful unit is accepted work. That also changes how you estimate savings. Count setup, review, exceptions, and maintenance alongside the time spent executing a successful task. A workflow that saves twenty minutes but requires frequent rescue needs a different business case from one that runs reliably.
For website owners, the same issue runs in reverse. Decide separately which automated activities help your business: discovery, content collection, account access, or transactions. Then check that your site’s controls reflect those decisions.
AI agents will keep improving. Their usefulness in a business will also depend on the relationships and permissions around them.
Before you ask how much work an agent can do, ask whether it can reach the work—and how you’ll know when it cannot.
Frequently asked questions
Why can I open a website but my AI agent cannot?
The requests may use different sessions, permissions, networks, or browser behavior. A site can also restrict automated clients. A failed request alone does not identify which condition caused the failure.
Does robots.txt control everything an agent can access?
No. It communicates crawler preferences. Authentication, application permissions, and enforcement such as firewall rules can separately affect whether a request succeeds.
Does an API or MCP connector guarantee reliable access?
No. It provides a supported route only for the operations, permissions, and service conditions it covers. Test the complete workflow, its limits, and the handoff when a step fails.
Sources and scope
Research checked October 21, 2026. The X posts are June 2026 context; the Amazon and Perplexity statements are from November 2025. We do not assert a current court outcome. Our October 20 retrieval observation is a single tool comparison, not a controlled access study.
- Amazon’s public position on Comet and Perplexity’s response.
- Cloudflare bot classifications, crawler management, and Pay Per Crawl documentation.
- TechTarget’s October 9 reporting.
Need to know whether an agent will work with your actual systems? Talk to OneWave about testing one workflow, including its access limits and exception handling.