No, we’re not talking about dystopian future robots taking over the world. We’re talking about plain text files that most websites do (or should!) have.
Fun fact: robots.txt was given its name because it’s a set of instructions for web crawlers, colloquially referred to as ‘robots’.
So, what is it?
As Google describes, the robots.txt is a plain file stored in your server that provides instructions for web crawlers. That’s all it is. You can generally find it on any website by typing the site’s domain name followed by /robots.txt. For example: https://upperdog.co.uk/robots.txt. It’s there to direct search engines and bots what to crawl and what not to crawl on a website.
There’s been a growing trend in providing AI agent-specific rules, something which is hotly contested. Google’s John Mueller himself said the content signals in robots.txt has “no effect whatsoever” on any web crawlers or LLMs, as noted in Reddit.
Robots.txt files are normally auto-generated by content management systems like WordPress, Wix, SquareSpace, Shopify, Woocommerce, Craft, etc. Usually, these files are already pre-configured and ready to go right out-of-the-box. However, most content management systems also allow users to make changes if needed. In WordPress, you can use various SEO plugins like Yoast or Rank Math to configure your robots.txt settings to your heart’s content.
Is Google expanding robots rules?
In a recent episode of Search Off the Record, Martin Splitt and Gary Illyes (two of Google’s search team advocates) spoke about the possibility of expanding robots.txt rules.
If you didn’t know, the rules are pretty limited. As of writing this, Google supports only the following robots.txt rules: User-agent, disallow, allow, & sitemap.
| Robots.txt rule | Description |
|---|---|
| User-agent | Specifies what user agents your rules apply to. There are countless user agents. The most commonly blocked user-agents include ahrefs bot and Screaming Frog SEO Spider. Likely for competitive reasons.
Normally, the most common rule is to denote a * which signifies all user agents. |
| allow | Not used very frequently, but it tells web crawlers explicitly what can be crawled. Useful if you disallow an entire subdirectory in your website and there’s one sub folder that you specifically need crawled. |
| disallow | The main rule that most websites make use of. It tells web crawlers specifically what not to crawl.
Note that if even if a webpage is indexed and you disallow it from being crawled, that webpage will remain indexed, because you’re preventing web crawlers from accessing it and seeing your noindex directive. Just an FYI. |
| sitemap | This helps web crawlers find your sitemap. Most CMS will include it in your robots.txt file by default, like Shopify or WordPress (if you use a plugin like Yoast or Rankmath). |
In Search Off the Record, Martin and Gary discussed the possibility of expanding the ruleset. This comes after a study using HTTP Archive and BigQuery, where millions of robots.txt files were analysed between 2025-2026.
They looked at stats like the most popular user-agents and directives taken from the study to gauge how people are using these files. Why? The team are considering plans to expand the rule set. In the words of Gary Illyes, they are considering the “top 10 or top 15 tags and add the ones” that they don’t currently support.
This is potentially big news for SEOs. And, if developers and SEOs start adopting LLMs.txt files at scale, there could be a very real possibility that Google will listen to the masses and add official support for these file types also. What if, alongside the sitemap, we added reference to an llms.txt file? Google might then comply. It’s worth trying.
What the HTTP Archive data shows
The data of millions of websites is publicly accessible in HTTP Archive, so naturally we put on our explorer hats and dug deep into the dataset.
Methodology
To get our findings we queried the HTTP Archive dataset, including the robots.txt stats which were first collected in February, 2025. The data we collected is based on the number of websites crawled in January 2026 for an updated view.
Because the dataset is so massive (over 1 million pages tracked), we’ve sampled the data to establish a reasonable representation. See our findings below for some interesting statistics.
Most popular robots.txt directives
According to the HTTP archive, most sites use crawl-delay as a directive. While Google does not support this, other search engines like Bing and Yandex apparently do.
What’s crawl-delay for? It tells crawlers to wait a specific time interval between requests, to help reduce the load on a server.
There’s no valid reason to use crawl-delay unless your site is getting hammered with requests. However, this is not an issue for Googlebot as it automatically adjusts its crawl rate depending on how servers respond. For instance, if a server returns an abundance of 500 http status codes, it will throttle its crawl requests until the server returns to normal.
Noindex directives in robots.txt
Along with crawl-delay, the most popular Google non-supported directive in robots.txt is ‘noindex’, which is deprecated since 2019. This is likely a result of webmasters neglecting their technical SEO over the years. If your site still uses the noindex directive, this is your cue to remove it!
If you want to stop a page from appearing in search results, use the proper <meta name=”robots” content=”noindex”> tag in your page’s <head>.
How many sites exceed Google’s 500KiB robots.txt limit?
According to the HHTP Archive dataset, 8% of websites exceed Google’s 500 kibibytes robots.txt limit.
Yes, Google has a strict cut off limit for robots.txt files; any content beyond the 500KiB threshold is duly ignored. This means 8% of websites include instructions that are completely ignored. Similarly, Google have recently updated their crawl documentation, stating that Googlebot only indexes the first 2MB of a page’s file types (HTML, CSS, JavaScript). PDFs have a lot more generous limit of 64MB as these don’t require the costly process of rendering.
What is the best approach for robots.txt?
The best approach is to make appropriate use of wildcards and avoid unnecessary lines of specific directives. Instead of page-specific rules, block sub-directories. Blocking individual pages is what noindex tags are for. This will consolidate your robots.txt file and ease maintenance.
Crawl delay – what bots are being throttled?
| User Agent | No. of sites using crawl delay | % |
|---|---|---|
| * (all user agents) | 854,961 | 47.3% |
| adsbot-google | 119,503 | 6.6% |
| ahrefsbot | 114,014 | 6.3% |
| mj12bot | 93,356 | 5.2% |
| gptbot | 67,961 | 3.8% |
| nutch | 64,006 | 3.5% |
| ahrefssiteaudit | 59,071 | 3.3% |
| 59,065 | 3.3% | |
| claudebot | 56,681 | 3.1% |
| ccbot | 55,804 | 3.1% |
| dotbot | 53,037 | 2.9% |
| adsbot-google-mobile | 52,919 | 2.9% |
| amazonbot | 52,709 | 2.9% |
| google-extended | 52,581 | 2.9% |
| bytespider | 51,623 | 2.9% |
| TOTAL | 1,807,291 |
According to the HTTP archive data, 47% of robots.txt files include crawl-delay rules targeting all user agents. The wildcard * dominates, which is expected. Ironically, Googlebot ignores crawl-delay, so this rule is completely ignored.
Notably, users have started referencing AI bots in their robots.txt files. GPTBot (3.8%), ClaudeBot (3.1%) and AmazonBot (2.9%) being the most prominent These are relatively new bots as well, and we expect this trend to continue growing as site owners continue to monitor AI activity on their sites.
How many sites contain broken robots.txt files?
| Status Code | sites | % |
|---|---|---|
| 200 | 94,191 | 85.7% |
| 404 | 13,396 | 12.2% |
| Unspecified | 1,030 | 0.9% |
| 403 | 896 | 0.8% |
| 500 | 111 | 0.1% |
| 429 | 95 | 0.1% |
We queried the HTTP archive data and found that, since January 2026, approximately 12% of websites’ robots.txt files returned 404s. So approximately 86% of robots.txt files are accessible, which is a surprising number.
Examples of weird and wonderful robots.txt
As a bit of fun trivia, we wanted to share some of the most bizzarre robots.txt files out in the wild. For fellow SEO enthusiasts, coming across one of these feels almost as exciting as catching Pokemon.
Johnmu.com
https://johnmu.com/robots.txt
John Mueller is the senior search analyst at Google and is known for his quips on social media, as well as frequent appearances in SEO events and podcasts. His comedic flair shines through in this robots.txt example on his own site, which is curiously a .wav file voice-recorded by the legend himself!
Garryillyes.com
https://garyillyes.com/robots.txt
Just like Mueller, Gary Illyes is also a Google search analyst. He’s also known for his quirky humour and the Podcast “Search Off the Record”. In this example, Gary’s tongue-in-cheek humour is unmistakable. Upon loading the webpage, the user is greeted with the 2010s’s viral YouTube video – Rick Astley’s Never Gonna Give You Up. And if that’s not enough, there’s an elusive “click me” button that escapes every time the user attempts to hover the cursor over it.
Wayfair.co.uk
Wayfair has always been very proactive with their SEO, this appears to be evident when you look at their robots.txt file.
What’s interesting about Wayfair’s robots.txt file is just how long it is – there’s 74 lines! I can’t imagine how much of a headache it is to maintain and review it quarterly. The txt file contains 11 Allow and 55 Disallow rules, not in any logical order either! Whoever’s in charge of tech SEO at Wayfair: are you ok? Someone please check up on this poor soul.
Tamethebots.co.uk – “The Lord of the Robots.txt”
The site owner of this glorious website is Dave Smart, a long-standing member of the SEO community and tech SEO Brighton speaker. Dave Smart is known for his wacky SEO experiments and contributions to the community. It is no wonder then, that his website, tamethebots, contains a rather unusual and epic robots.txt that resonates with fans of Lord of The Rings.
Commented at the very top of the txt file, encased in an ASCII art scroll, is inscribed the following:
Three Rules for the Elven-kings under the sky,
Seven for the Dwarf-lords in their halls of stone,
Nine for Mortal Men doomed to die,
One for the Dark Lord on his dark throne
In the Land of Mordor where the Shadows lie.
One Rule to block them all,
One Rule to crawl them,
One Rule to bring them all and in the darkness bind them
In the Land of Mordor where the Shadows lie.
That’s not the best bit, though, notice the declared user agents: Gil-galad, Elrond and Galadriel – they are all allowed to crawl the site, whereas Sauron and Saruman are disallowed! This is quite possibly the coolest robots.txt file in existence. There is one villain Dave forgot to block though: Shelob. Why? Because she’s a spider crawling the web. Ok, I’ll see myself out.
If you’re interested in a receiving a robots.txt audit along with an in-depth tech SEO report, get in contact with us. Upperdog is a leading SEO and GEO agency, helping brands combine traditional SEO and GEO strategies to move from being visible to being selected.
If you want your content to show up where decisions are really being made, get in touch by emailing hello@upperdog.co.uk or calling 01202 798820. We’d love to help you turn visibility into real influence.
Sources:
https://developers.google.com/search/blog/2025/02/intro-robots-refresher
https://search-off-the-record.libsyn.com/analysing-robotstxt-at-scale-with-http-archive-and-bigquery
https://almanac.httparchive.org/en/2025/
https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers
