Google has completely reorganized its crawlers and user-triggered fetchers documentation. It used to be all on one page and now it is in several pages. Most of the changes were just moving content around but Google did add sections for what product each crawler affects, and added a robots.txt snippet for each crawler to demonstrate how to use the user agent tokens.
from Search Engine Roundtable https://ift.tt/BbuqJl6
No comments:
Post a Comment
Leave Post