Skip to main contentFree scan: how AI-ready is your website?Start now →
Back to Resources

Allowing BrandHalo crawlers through your firewall

·5 min read

BrandHalo crawls your site for compliance scans and for AI readiness checks at brandhalo.io/aiready. If a firewall or web application firewall (WAF) blocks those requests, scans can fail or come back incomplete, including HTTP 403 responses and security challenge pages.

Add the User-Agent values below to your firewall allow list so scans can reach your homepage and site files.

User-Agent strings to allow

Copy the full string into an exact-match rule. If your firewall matches part of the header, allow BrandHalo-Crawler and BrandHalo-AIChecker.

  • BrandHalo-Crawler/1.0 (+https://brandhalo.io) - channel crawls and compliance scans
  • BrandHalo-AIChecker/1.0 (+https://brandhalo.io) - AI readiness scans (brandhalo.io/aiready)

Firewall and WAF rules

Create an allow rule for those two User-Agent values. Set the action to "Allow" or "Skip" (skip remaining rules) so the crawlers can reach your site.

Cloudflare WAF: quick setup

If you use Cloudflare, add a Custom rule with action "Skip" so BrandHalo crawlers can reach your site. Use quick add to open a prefilled rule in your dashboard, then choose which WAF components to skip and deploy.

Rule name

Allow BrandHalo crawlers

Rule expression

(http.user_agent contains "BrandHalo-AIChecker") or (http.user_agent contains "BrandHalo-Crawler")
Add rule to Cloudflare

The expression matches these full User-Agent values:

  • BrandHalo-Crawler/1.0 (+https://brandhalo.io)
  • BrandHalo-AIChecker/1.0 (+https://brandhalo.io)

After Cloudflare opens, sign in, pick your domain, review the prefilled rule, select Skip targets (for example all remaining custom rules and Super Bot Fight Mode rules), then deploy.

Other firewalls

AWS WAF, Sucuri, Imperva, and similar tools use the same match. Allow requests whose User-Agent contains either full string above.

IP allow listing

BrandHalo does not use a fixed set of IP addresses, so an IP allow list will not stay reliable. Allowing the User-Agent strings above is the approach to use. If you need IP allow listing on an Enterprise plan with dedicated egress IPs, contact us at hello@brandhalo.io.

After updating your firewall

Once your exemptions are in place, run another scan at brandhalo.io/aiready to confirm we can reach your homepage. For ongoing monitoring, compliance scans, and llms.txt generation, create a BrandHalo account and add your site as a channel.

Related articles

Part of BrandHalo's resource library on brand governance, connectors, Brand Agent, and keeping your brand consistent across AI tools and channels.

View all articles

Ready to govern your brand?

BrandHalo gives you the infrastructure to centralise, monitor, and protect your brand across every channel and AI touchpoint.

You've read the guide. Now put it into practice.

Allowing BrandHalo crawlers through your firewall