robots.txt: disallow access to snapshots

My dmesg is filled with the oom killer bringing down processes while the
Bingbot downloads every snapshot for every commit of the Linux kernel in
tar.xz format. Sure, I should be running with memory limits, and now I'm
using cgroups, but a more general solution is to prevent crawlers from
wasting resources like that in the first place.

Suggested-by: Natanael Copa <ncopa@alpinelinux.org>
Suggested-by: Julius Plenz <plenz@cis.fu-berlin.de>
Signed-off-by: Jason A. Donenfeld <Jason@zx2c4.com>

diff --git a/Makefile b/Makefile
index 00b32690d4314b3353f52a3ae3aa84b798a4e72d..f11b60f702a166142fe5144a3f75cade62a26fa3 100644 (file)
--- a/Makefile
+++ b/Makefile
@@ -78,6 +78,7 @@ install: all
        $(INSTALL) -m 0644 cgit.css $(DESTDIR)$(CGIT_DATA_PATH)/cgit.css
        $(INSTALL) -m 0644 cgit.png $(DESTDIR)$(CGIT_DATA_PATH)/cgit.png
        $(INSTALL) -m 0644 favicon.ico $(DESTDIR)$(CGIT_DATA_PATH)/favicon.ico
+       $(INSTALL) -m 0644 robots.txt $(DESTDIR)$(CGIT_DATA_PATH)/robots.txt
        $(INSTALL) -m 0755 -d $(DESTDIR)$(filterdir)
        $(COPYTREE)  filters/* $(DESTDIR)$(filterdir)
 

diff --git a/robots.txt b/robots.txt
new file mode 100644 (file)
index 0000000..4ce948f
--- /dev/null
+++ b/robots.txt
@@ -0,0 +1,3 @@
+User-agent: *
+Disallow: /*/snapshot/*
+Allow: /