If you found this page in your server logs, PhareBot fetched a few pages of your website. Here is exactly what it takes, and how to stop it.
Add this to your robots.txt. We read it before every fetch and obey it. There is nothing to email and nobody to convince.
User-agent: PhareBot
Disallow: /Our requests carry PhareBot in the User-Agent header, along with a way to reach a human. If a page you needed us to skip was already fetched, the same rule stops it being used, and it drops out on the next run.
These numbers are read from the crawler's own configuration when this page is built, not written by hand. If the crawler changes and this page doesn't, the build fails.
The homepage, plus pricing, product and about pages found from your sitemap or your homepage links. That is the entire corpus we compare model claims against.
A single request per domain per interval, never in parallel. You should not be able to notice us in your traffic graphs.
The same site is not fetched again inside this window. The corpus is cheap but it is not free, and repeatedly hitting someone's site is how a bot gets blocked.
Declare a long crawl-delay in robots.txt and we cut the number of pages rather than ignoring the delay and taking the lot.
/equipe or /team page is named individuals — personal data we have no need for. Excluded by URL before the fetch, not discarded after. Job listings too.Disallow is a finding, not an obstacle. Anything we could not verify comes out as "couldn't verify".