About our crawler

SerpKeeper is a free tool that checks websites for pages that break Google's spam policies. When someone runs a check, our crawler visits that website's pages the way a search engine would.

How to recognise it

It identifies itself with this user agent:

Mozilla/5.0 (compatible; SerpKeeperBot/1.0; +https://www.serpkeeper.com/bot)

It does not pretend to be Googlebot. For one small test (comparing what search engines and phones are shown) it fetches up to 15 pages with Googlebot's and a phone browser's user agent, and says so in the report.

What it does

• It follows your robots.txt rules.
• It reads HTML pages only. It skips images, videos, fonts and downloads.
• It fetches a limited number of pages per check (up to 2,000) and stops after 180 minutes.
• It never submits forms or logs in.

How to block it

Add this to your robots.txt:

User-agent: SerpKeeperBot
Disallow: /

To have your domain blocked completely, even for people who can prove they own it, send us an opt-out request.