9 – Standard files
A handful of files are expected in fixed locations on a website. They cost almost nothing to put there and are missing surprisingly often.
What is tested?
robots.txt
- That the file exists and can be fetched
- That it contains what it should
- That it points to the website's sitemap
Sitemap
- That a valid sitemap exists
- That it contains something
- That the addresses in
<loc>use the same domain as robots.txt - That the addresses start with
https:// - That no address appears more than once
- That it lists web pages and not other content
- That it doesn't exceed 50,000 entries
security.txt
- That the file exists at
/.well-known/security.txt - That the content is correctly formatted
- That the mandatory contact information is present
- That an expiry date is given, which the standard has required since draft 10
Feeds
- That RSS or Atom feeds are declared in the page metadata, if the site has any
How it affects the score
Each deviation counts as an issue at its stated severity. Poor sitemap hygiene, such as duplicates or mixed domains, lowers the score even when the file itself can be fetched.
Running the test
python default.py -u https://example.com -t 9 -r