Add sitemap index and serve sitemap for subprojects

[cedk]

The sitemap index lists all the sitemap locations for a project including the subprojects.
The sitemap of subprojects are also served from the parent domain.

Closes #6841

---

📚 Documentation previews 📚

• User's documentation (docs): https://docs--10629.org.readthedocs.build/en/10629/

• Developer's documentation (dev): https://dev--10629.org.readthedocs.build/en/10629/

[ericholscher]

Is there any more info on what the sitemap_index is for? I'm not super familiar with this code, but it seems like a nice addition!

[cedk]

The sitemap index is a file to list sitemaps of a website: https://en.wikipedia.org/wiki/Sitemaps#File_format
So the idea is that when you use subproject, you want to have a single sitemap file that list all the sitemaps so search engine can discover all the pages.

[cedk]

I'm wondering if the robots.txt should refer to the sitemap index instead of the project sitemap. Such that all projects with subproject will gain sitemap discoverability of subproject automatically without having to declare the sitemap index to the search engine.

[cedk]

How to relaunch the tests?

[humitos]

I'm not sure what happened here. Maybe it's a CircleCI configuration that doesn't allow to trigger test on contributor's PRs? 🤷🏼

[minaelee]

Hello RTD team! Is anyone able to provide an update on this PR and the CircleCI test failure? This would be a welcome addition.

[humitos]

@minaelee If I understand correctly, this PR is blocked on more discussion about the "Sitemap index" concept. We are not familiar with it and the Wikipedia link shared does not provide enough context about how it works, how search engine use it (or even if they respect it (?)) and more.

If you are using Sphinx, I've been recommending the Sphinx extension https://sphinx-sitemap.readthedocs.io/en/latest/ that allows users to customize sitemaps with their needs. Let me know if this helps.