PhareBot

If you found this page in your server logs, PhareBot fetched a few pages of your website. Here is exactly what it takes, and how to stop it.

How to make it stop

Add this to your robots.txt. We read it before every fetch and obey it. There is nothing to email and nobody to convince.

User-agent: PhareBot
Disallow: /

Our requests carry PhareBot in the User-Agent header, along with a way to reach a human. If a page you needed us to skip was already fetched, the same rule stops it being used, and it drops out on the next run.

What it actually does

These numbers are read from the crawler's own configuration when this page is built, not written by hand. If the crawler changes and this page doesn't, the build fails.

  • 15 pages

    We read a handful of pages, not your whole site

    The homepage, plus pricing, product and about pages found from your sitemap or your homepage links. That is the entire corpus we compare model claims against.

  • 2s

    One request at a time, spaced out

    A single request per domain per interval, never in parallel. You should not be able to notice us in your traffic graphs.

  • 30 days

    We come back rarely

    The same site is not fetched again inside this window. The corpus is cheap but it is not free, and repeatedly hitting someone's site is how a bot gets blocked.

  • 5 pages

    If you ask us to slow down, we take less

    Declare a long crawl-delay in robots.txt and we cut the number of pages rather than ignoring the delay and taking the lot.

request timeout
10s
response size cap
2 MB

What it never does

  • It skips your team pages. An/equipe or /team page is named individuals — personal data we have no need for. Excluded by URL before the fetch, not discarded after. Job listings too.
  • It does not run JavaScript. No headless browser. If a page needs one, we record that we could not read it and the report says so, rather than working around it.
  • It does not route around a block. A 403, a bot challenge or a Disallow is a finding, not an obstacle. Anything we could not verify comes out as "couldn't verify".
  • It only ever fetches your own domain. Never a competitor's, never a third party's, in this step.
  • It collects nothing about your visitors. We read published pages. No logs, no analytics, no personal data.