.Google.com's John Mueller responded to a concern about why Google.com marks pages that are actually prohibited from creeping through robots.txt and why the it is actually risk-free to neglect the relevant Browse Console files concerning those creeps.Robot Visitor Traffic To Query Criterion URLs.The individual inquiring the inquiry documented that bots were generating web links to non-existent inquiry parameter Links (? q= xyz) to webpages with noindex meta tags that are likewise shut out in robots.txt. What prompted the inquiry is that Google is actually crawling the web links to those pages, receiving blocked out through robots.txt (without watching a noindex robots meta tag) after that receiving turned up in Google.com Browse Console as "Indexed, though blocked out by robots.txt.".The person inquired the following inquiry:." However listed below is actually the big inquiry: why would certainly Google mark web pages when they can't even view the material? What is actually the perk in that?".Google.com's John Mueller verified that if they can't creep the web page they can't find the noindex meta tag. He likewise helps make an interesting reference of the website: search operator, recommending to disregard the results because the "ordinary" individuals won't find those results.He created:." Yes, you're proper: if our experts can't creep the page, our experts can't observe the noindex. That claimed, if our team can't crawl the webpages, then there is actually certainly not a whole lot for our company to index. Thus while you may see some of those web pages along with a targeted web site:- query, the normal user won't view all of them, so I definitely would not fuss over it. Noindex is likewise great (without robots.txt disallow), it simply implies the URLs will wind up being crept (as well as find yourself in the Explore Console file for crawled/not recorded-- neither of these statuses create concerns to the remainder of the web site). The vital part is that you don't create all of them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the limitations in using the Web site: hunt accelerated hunt operator for analysis factors. One of those factors is given that it's certainly not linked to the routine hunt mark, it is actually a separate thing entirely.Google.com's John Mueller commented on the web site hunt driver in 2021:." The brief solution is actually that a site: query is certainly not meant to be full, neither made use of for diagnostics functions.A web site concern is actually a details type of search that limits the outcomes to a specific site. It's basically only words internet site, a colon, and afterwards the website's domain.This question limits the end results to a specific site. It is actually certainly not meant to be an extensive selection of all the web pages coming from that web site.".2. Noindex tag without using a robots.txt is actually fine for these sort of scenarios where a robot is connecting to non-existent pages that are actually getting discovered through Googlebot.3. Links with the noindex tag are going to produce a "crawled/not recorded" entry in Explore Console and also those will not possess an adverse effect on the remainder of the site.Go through the concern and also respond to on LinkedIn:.Why will Google index web pages when they can not even view the web content?Featured Image through Shutterstock/Krakenimages. com.