Home / SpotgrowAI
SpotgrowAI
Meet SpotgrowAIBot
Our crawler reads public pages to analyse the content and technical health of websites added to SpotgrowAI.
Identification and control
Crawler requests identify themselves as SpotgrowAIBot/1.0. You can use robots.txt directives to indicate which paths should not be crawled.
User-agent: SpotgrowAIBot Disallow: /private/
If you notice an issue with a crawler visit, send us the domain, date and affected paths.
Contact the teamFind the next step for your project.