meritkollen/html/robots.txt

User-agent: *
Disallow: /assets/
Disallow: /include/
Disallow: /404.html
Disallow: /50x.html
Disallow: /sw.js
Disallow: /cache-polyfill.js
Disallow: /ads.txt
upload from server uploaded all websites as they are on the server 2019-09-17 21:20:48 +02:00			`User-agent: *`
			`Disallow: /assets/`
			`Disallow: /include/`
			`Disallow: /404.html`
update meta descriptions and robots 2019-11-06 20:32:18 +01:00			`Disallow: /50x.html`
			`Disallow: /sw.js`
			`Disallow: /cache-polyfill.js`
			`Disallow: /ads.txt`