Answers / Scraping & robots.txt
What does robots.txt mean for a scraper?
robots.txt is a file at a site's root that tells automated crawlers which paths they may or may not fetch, by user-agent. It's a widely-followed convention, not a law, and it governs access, not what you may do with the content. Respecting it is good practice, but it's only one of several signals that decide whether a use is allowed.
Was this answer helpful?
AIScrapeSafe answers reflect detected signals and published rules — not legal advice.