Google Adds More Details To Robots.txt Specifications Help Document

Google has added a bunch of new supporting details to the robots.txt specification document. Jamie Indigo noticed these changes and explained what was added include "explicit details on the files for IDNs, IP-addresses, and port numbered hostnames."

from Search Engine Roundtable https://ift.tt/3ntZACf