NoindexedBot

An SEO watchdog acting for a customer of noindexed, the owner or operator of the site being checked. Not a research crawler, not a scraper.

Customer-designated URLs

Only URLs the site owner registered as critical, typically fewer than 25, plus /robots.txt. No JS rendering.

At most every 15 minutes, short list only

The critical list is fetched at most every 15 minutes (hourly or daily on lower plans), or immediately after the owner’s deploy hook fires.

Once daily

sitemap.xml, plus pages discovered from it and from same-host links — up to the plan limit — are checked once daily, around 04:00 UTC. The bot never leaves the owner’s own host.

robots.txt

The bot reads robots.txt as monitored data, to alert the owner when it changes. It does not apply it as crawl rules: every URL on the list was explicitly submitted by the site’s owner, so listed URLs are fetched even where robots.txt disallows crawling.