Bot accessibility testing: three layers, two questions
How to verify crawlers can read your catalog and are blocked from cart and checkout: robots.txt, firewall and server fail differently, so all three layers need testing.
Founder of CookingMetrics · creator of IndexNow Connect
Data analytics and technical SEO consultant. Founder of CookingMetrics (Data Analytics Partners) and creator of IndexNow Connect. Writes about indexing, Cloudflare and GEO from real cases of stores connected to the product — including the ones that went wrong and what they taught us.
How to verify crawlers can read your catalog and are blocked from cart and checkout: robots.txt, firewall and server fail differently, so all three layers need testing.
Your platform writes robots.txt and Cloudflare rewrites it. How the editor works: it reads your real file, one switch per bot, and publishes the result through a Worker.
The robots.txt directive Cloudflare created in 2025: how to write it, what search, ai-input and ai-train mean, and why it declares use without blocking any access.
GEO explained without the hype: how AI engines pick their sources, the 5 levers an online store can pull, and how to measure whether ChatGPT cites you.
What actually matters in your sitemap: honest lastmod, the changefreq and priority myths, Bing's daily re-reads, and why freshness now feeds AI-generated answers.
Over half of web traffic is bots. If AI crawlers can't read your store, you don't exist in ChatGPT. Audit it in Cloudflare in 15 minutes — no redesign needed.
DNS, TLS, cache, Bot Fight Mode and AI Crawl Control: the exact Cloudflare settings that decide whether Googlebot, Bingbot and AI search bots can see your store.
Put your store behind Cloudflare in one afternoon, free and without code: 5 plain-language steps and the 3 mistakes that can make you invisible on Google and ChatGPT.
How to serve the {key}.txt file at the root of a SaaS-hosted domain: free Workers, key-agnostic routes and the real case of A records bypassing your zone.
Microsoft Bing's push protocol explained: how the key works, who participates, what it does not do, and why Bing's index is the gateway to ChatGPT and Copilot.