About our crawler

What it does

• Identifies itself honestly as BlueChipGarageBot with a link to this page. It does not pretend to be a browser.
• Reads and obeys robots.txt on every host, and re-checks it before every path. A disallowed path is skipped and logged as skipped.
• Rate-limits itself per host — by default one request every few seconds, and slower where a Crawl-delay is published.
• Sends conditional requests and caches responses on disk, so we re-parse our own copy rather than re-fetching yours.
• Backs off on 429 and 5xx responses, honouring Retry-After, and disables a source entirely after repeated failures.
• Caps the number of pages any single run may fetch.

What we store

For most sources we keep only facts about the vehicle — year, model, mileage, what it sold for and when — together with a link back to your listing. Facts are not copyrightable; photographs and listing prose are, so for those sources we do not copy them at all. We never create accounts on sites we collect from, and we never use collected content to train models.