I think a better design would be to define a spec for search data so sites that implement it could generate a ".well-known/search_data.zst" (or whatever) and people would only need to crawl the site to check compliance with the spec.