About this crawler

cf-scrape-monitor is a small, non-commercial monitoring project. If you see the User-Agent cf-scrape-monitor/0.1 in your logs, the request came from here.

What it fetches

• books.toscrape.com: the first 3 catalogue pages (title, price, availability, rating)
• quotes.toscrape.com: the first page (quote text, author, tags)

Both sites exist for scraping practice.

How it behaves

• Runs at most every 6 hours (at minute 17).
• Reads and obeys robots.txt, re-checked every 24 hours.
• Sends one request at a time per host, with a 1.5 s pause between requests.
• Uses conditional requests (If-None-Match / If-Modified-Since), so an unchanged page costs only a 304.
• Backs off on 429 and 5xx, honors Retry-After, and retries at most once.
• Collects no personal data.

Opting out

Disallow cf-scrape-monitor in your robots.txt. The crawler re-reads it at least every 24 hours and skips any disallowed path.