Robort.txt Generator is a free tool from Digital Scholar which allows Robots.txt files to specify how search engines should crawl web pages. Meta's robots.txt file instructs Googlebot and other crawler robots not to index or follow anything but the home page.