NoindexedBot

An SEO watchdog acting for a customer of noindexed, the owner or operator of the site being checked. Not a research crawler, not a scraper.

Customer-designated URLs

Only URLs the site owner registered as critical, typically fewer than 25, plus /robots.txt. No JS rendering, link following, or site-wide crawl.

Every 15 minutes

The critical list is fetched every 15 minutes, or immediately after the owner’s deploy hook fires.

Once daily

sitemap.xml and pages up to the plan limit are checked once daily, around 04:00 UTC.

robots.txt

The bot reads robots.txt as monitored data, to alert the owner when it changes. It does not apply it as crawl rules: every URL on the list was explicitly submitted by the site’s owner, so listed URLs are fetched even where robots.txt disallows crawling.