robots.txt: add more git.sr.ht routes

Most crawlers end up crawling the entire history of a repository, which is kind of pointless and expensive.
2022-05-25 16:43:04 +02:00 · 2022-05-25 16:43:04 +02:00 · a195e2fc77
parent 31237ce40d
commit a195e2fc77
1 changed files with 2 additions and 0 deletions
--- a/robots.txt
+++ b/robots.txt
@ -18,6 +18,8 @@ Disallow: /*.tar.gz$
 Disallow: /metrics
 Disallow: /*/*/blame/*
 Disallow: /*/*/log/*
+Disallow: /*/*/tree/*
+Disallow: /*/*/item/*

 # Too aggressive, marketing/SEO
 User-agent: SemrushBot