User-agent: SemrushBot Disallow: / User-agent: Yandex Disallow: / # blocks access to the whole site User-agent: * Disallow: /bin/ Disallow: /lib/ Disallow: /setup/ Disallow: /site/ Disallow: /var/