Whitelisting the Rankability crawler

How to whitelist Rankability’s crawler so it can access your website for audits, content research, and knowledge base imports. Includes the exact user-agent string and step-by-step instructions for popular firewalls and CDNs.

Rankability crawls websites to import knowledge base sources, run site audits, and research content for the Copywriter. Some websites use firewalls, CDNs, or bot-protection services that block automated requests. If your website blocks the crawler, sources will fail with a bot-protection or 403/429 error.

Rankability’s user-agent

Every request from Rankability’s crawler uses this user-agent string:

Mozilla/5.0 (compatible; Rankability/1.0)

This user-agent is the same for all Rankability features — Serena source imports, Copywriter research, site audits, and domain discovery. It does not change based on your agency name or branding settings.

When do I need to whitelist?

You need to whitelist the crawler if:

• A knowledge base source shows a bot-protection error or 403 Forbidden status.
• A URL or Crawl source import fails with a message mentioning “bot protection” or “access denied.”
• A client website scan says “couldn’t automatically discover pages” and the site uses Cloudflare, Sucuri, Akamai, or a similar service.
• The Copywriter’s SERP research cannot fetch a competitor page behind bot protection.

If your website is publicly accessible without bot protection, no action is needed.

How to whitelist

The exact steps depend on your firewall or CDN provider. Below are instructions for the most common services.

Cloudflare

1. Log in to the Cloudflare dashboard and select your domain.
2. Go to Security → WAF (Web Application Firewall).
3. Click Create rule.
4. Set the field to User Agent, the operator to contains, and the value to Rankability/1.0.
5. Set the action to Skip (or Allow).
6. Save the rule.

Sucuri

1. Log in to the Sucuri dashboard.
2. Go to Access Control → User-Agent Filtering.
3. Under the Allowlist section, add Rankability/1.0.
4. Save changes.

Akamai

1. Open Akamai Control Center.
2. Navigate to your property’s Security Configuration.
3. Add a custom rule that matches the User-Agent header containing Rankability/1.0 and sets the action to Allow.
4. Activate the configuration.

AWS WAF

1. Open the AWS WAF console.
2. Select your Web ACL.
3. Create a new rule with a String match condition on the User-Agent header containing Rankability/1.0.
4. Set the rule action to Allow.
5. Save and deploy the updated Web ACL.

Nginx (server-level)

If you manage your own Nginx server and have bot-blocking rules, add an exception before the block:

if ($http_user_agent ~* "Rankability/1.0") {
  set $block_bot 0;
}

Apache (.htaccess)

If your .htaccess blocks bots by user-agent, add this before the deny rules:

RewriteCond %{HTTP_USER_AGENT} Rankability/1.0
RewriteRule .* - [L]

Other providers

For any firewall or CDN not listed above, look for a “User-Agent allowlist” or “Bot management” setting and add Rankability/1.0 as an allowed user-agent string.

Where to find the user-agent in Rankability

You can also find the user-agent string inside Rankability:

1. Open any client and go to Ask Serena.
2. Click + Add source.
3. Select the URL or Crawl tab.
4. The user-agent string is displayed in the info box at the top of the dialog.

Verifying the whitelist works

After adding the whitelist rule:

1. Go back to the failed source in Rankability.
2. Delete the failed source and re-add it.
3. The source should now import successfully.

If it still fails, double-check that the firewall rule is active and that there are no additional layers of bot protection (e.g., a hosting provider firewall on top of Cloudflare).

Related articles

• Using Serena chat — How to add sources and chat with Serena.
• Serena: Knowledge base and data connections — Managing knowledge base sources and data connections.
• Troubleshooting common issues — Fixes for common issues across all modules.