.Google's John Mueller addressed an inquiry about why Google.com indexes pages that are disallowed coming from crawling by robots.txt as well as why the it's secure to dismiss the related Explore Console files concerning those crawls.Bot Web Traffic To Question Criterion URLs.The individual asking the inquiry documented that robots were actually developing web links to non-existent inquiry specification URLs (? q= xyz) to webpages with noindex meta tags that are additionally blocked out in robots.txt. What motivated the concern is actually that Google is actually crawling the links to those pages, getting blocked by robots.txt (without noticing a noindex robotics meta tag) at that point receiving reported in Google Search Console as "Indexed, though obstructed through robots.txt.".The person talked to the complying with inquiry:." However right here's the significant question: why would certainly Google.com index webpages when they can not even view the material? What's the perk in that?".Google.com's John Mueller validated that if they can not crawl the web page they can't observe the noindex meta tag. He also creates an interesting acknowledgment of the internet site: hunt driver, encouraging to disregard the end results considering that the "typical" individuals won't see those end results.He wrote:." Yes, you are actually right: if our company can not crawl the page, our team can not view the noindex. That pointed out, if our company can't crawl the webpages, then there is actually not a great deal for us to mark. Thus while you might view several of those webpages along with a targeted site:- inquiry, the ordinary user won't see all of them, so I definitely would not fuss over it. Noindex is actually also great (without robots.txt disallow), it merely suggests the URLs are going to find yourself being crept (and also find yourself in the Explore Console record for crawled/not recorded-- neither of these conditions cause concerns to the remainder of the website). The integral part is actually that you do not create them crawlable + indexable.".Takeaways:.1. Mueller's response validates the limits in using the Internet site: search progressed hunt driver for diagnostic factors. Among those causes is since it is actually certainly not attached to the normal hunt mark, it's a distinct thing altogether.Google.com's John Mueller commented on the web site search operator in 2021:." The brief answer is actually that a site: concern is certainly not indicated to become complete, nor made use of for diagnostics reasons.A web site question is actually a specific type of hunt that restricts the outcomes to a particular website. It is actually essentially just words website, a bowel, and after that the internet site's domain.This query limits the outcomes to a details site. It's certainly not indicated to become a comprehensive assortment of all the pages from that site.".2. Noindex tag without making use of a robots.txt is actually fine for these sort of circumstances where a robot is actually connecting to non-existent web pages that are acquiring discovered by Googlebot.3. URLs with the noindex tag will produce a "crawled/not listed" item in Look Console which those will not possess an adverse effect on the rest of the website.Go through the concern and answer on LinkedIn:.Why would certainly Google mark pages when they can not even observe the information?Featured Photo by Shutterstock/Krakenimages. com.