> For the complete documentation index, see [llms.txt](https://help.rankability.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.rankability.com/troubleshooting/whitelisting-rankability-crawler.md).

# Whitelisting the Rankability crawler

How to whitelist Rankability’s crawler so it can access your website for audits, content research, and knowledge base imports. Includes the exact user-agent string and step-by-step instructions for popular firewalls and CDNs.

Rankability crawls websites to import knowledge base sources, run site audits, and research content for the Copywriter. Some websites use firewalls, CDNs, or bot-protection services that block automated requests. If your website blocks the crawler, sources will fail with a bot-protection or 403/429 error.

## Rankability’s user-agent

Every request from Rankability’s crawler uses this user-agent string:

```
Mozilla/5.0 (compatible; Rankability/1.0)
```

This user-agent is the same for all Rankability features — Serena source imports, Copywriter research, site audits, and domain discovery. It does not change based on your agency name or branding settings.

## When do I need to whitelist?

You need to whitelist the crawler if:

* A knowledge base source shows a **bot-protection error** or **403 Forbidden** status.
* A URL or Crawl source import fails with a message mentioning “bot protection” or “access denied.”
* A client website scan says “couldn’t automatically discover pages” and the site uses Cloudflare, Sucuri, Akamai, or a similar service.
* The Copywriter’s SERP research cannot fetch a competitor page behind bot protection.

If your website is publicly accessible without bot protection, no action is needed.

## How to whitelist

The exact steps depend on your firewall or CDN provider. Below are instructions for the most common services.

### Cloudflare

1. Log in to the **Cloudflare dashboard** and select your domain.
2. Go to **Security → WAF** (Web Application Firewall).
3. Click **Create rule**.
4. Set the field to **User Agent**, the operator to **contains**, and the value to `Rankability/1.0`.
5. Set the action to **Skip** (or **Allow**).
6. Save the rule.

### Sucuri

1. Log in to the **Sucuri dashboard**.
2. Go to **Access Control → User-Agent Filtering**.
3. Under the **Allowlist** section, add `Rankability/1.0`.
4. Save changes.

### Akamai

1. Open **Akamai Control Center**.
2. Navigate to your property’s **Security Configuration**.
3. Add a custom rule that matches the User-Agent header containing `Rankability/1.0` and sets the action to **Allow**.
4. Activate the configuration.

### AWS WAF

1. Open the **AWS WAF console**.
2. Select your Web ACL.
3. Create a new rule with a **String match condition** on the **User-Agent** header containing `Rankability/1.0`.
4. Set the rule action to **Allow**.
5. Save and deploy the updated Web ACL.

### Nginx (server-level)

If you manage your own Nginx server and have bot-blocking rules, add an exception before the block:

```
if ($http_user_agent ~* "Rankability/1.0") {
  set $block_bot 0;
}
```

### Apache (.htaccess)

If your `.htaccess` blocks bots by user-agent, add this before the deny rules:

```
RewriteCond %{HTTP_USER_AGENT} Rankability/1.0
RewriteRule .* - [L]
```

### Other providers

For any firewall or CDN not listed above, look for a “User-Agent allowlist” or “Bot management” setting and add `Rankability/1.0` as an allowed user-agent string.

## Where to find the user-agent in Rankability

You can also find the user-agent string inside Rankability:

1. Open any client and go to **Chat**.
2. Click **+ Add source**.
3. Select the **URL** or **Crawl** tab.
4. The user-agent string is displayed in the info box at the top of the dialog.

## Verifying the whitelist works

After adding the whitelist rule:

1. Go back to the failed source in Rankability.
2. Delete the failed source and re-add it.
3. The source should now import successfully.

If it still fails, double-check that the firewall rule is active and that there are no additional layers of bot protection (e.g., a hosting provider firewall on top of Cloudflare).

## Related articles

* [Using Serena chat](/serena/using-advisor.md) — How to add sources and chat with Serena.
* [Serena: Knowledge base and data connections](/serena/advisor-knowledge-base-and-data-connections.md) — Managing knowledge base sources and data connections.
* [Troubleshooting common issues](/troubleshooting/troubleshooting-common-issues.md) — Fixes for common issues across all modules.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://help.rankability.com/troubleshooting/whitelisting-rankability-crawler.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.