# robots.txt for www.asifood.com # Updated 2026-10-01. Blog pagination, author pages and PDFs are open so # search engines and AI answer engines can find every post and cite our guides. # # This single group covers every crawler, including AI search and answer # engines (OAI-SearchBot, ChatGPT-User, GPTBot, PerplexityBot, ClaudeBot, # Google-Extended, Applebot). Don't add a separate group for one bot unless # you copy these rules into it: a bot with its own group ignores this one. User-agent: * # HubSpot system pages and site search results Disallow: /_hcms/ Disallow: /hs/ Disallow: /hs-search-results # HubSpot preview and tracking parameters (duplicate URLs) Disallow: /*?*hs_preview= Disallow: /*?*hsCacheBuster= Disallow: /*?*hs_amp= # Thin blog tag pages Disallow: /*/tag/ # Form confirmation pages Disallow: /thank-you Disallow: /thanks Disallow: /ty- Disallow: /-ty # Site search Disallow: /search Disallow: /*?search= # Office files (templates and lead-magnet downloads). PDFs stay open. Disallow: /hubfs/*.xlsx Disallow: /hubfs/*.xls Disallow: /hubfs/*.docx Disallow: /hubfs/*.doc Disallow: /hubfs/*.pptx # Form-gated PDFs: block each one by name, for example # Disallow: /hubfs/your-gated-guide.pdf Sitemap: https://www.asifood.com/sitemap.xml Disallow: /_hcms/preview/ Disallow: /hs/manage-preferences/ Disallow: /hs/preferences-center/ Disallow: /*?*hs_preview=* Disallow: /*?*hsCacheBuster=*