Gary Illyes shared a nice little tidbit on LinkedIn about robots.txt files. He said that only a tiny number of robots.txt files are over 500 kilobytes. I mean, most robots.txt files have a few lines of text, so this makes sense but still, it is a nice tidbit of knowledge.
from Search Engine Roundtable https://ift.tt/GY7ncJm
No comments:
Post a Comment
Leave Post