Verified owners can ask the crawler to ignore robots.txt (useful when a staging or private section is blocked). Everyone else's checks always follow robots.txt.
robots.txt