Configure robots.txt
The robots.txt file tells search engines which parts of your website they can or cannot crawl. The Robots Text tab includes the following fields: User-agent: Specify the search engine crawler to which the rule applies. Disallow: Specify the website ...