Crawl Budget & Indexation

Master discovery/crawling/indexation: logs, robots.txt, sitemaps, pagination/facets, prioritization.