The first version of llms.txt I published was robots.txt with a different filename. It took a second commit, ten minutes later, to actually write something that resembled what the file is supposed to be.
A CSS preload optimization shipped, ran on every page load, and did nothing. Not "did nothing dramatic" — did nothing at all, silently, for the length of time it took someone to notice the preload tags weren't matching anything a browser would actually reuse. The fix was two characters in a regular expression.
No human being reads this site's sitemap.xml. It exists entirely for crawlers, gets zero direct visits worth counting, and if it silently broke tomorrow the only symptom would be a slow decline in indexing over weeks, with nothing in the site's own analytics pointing at the cause.
Drupal ships a sensible robots.txt. It blocks the admin paths, the login and registration forms, the search pages, and the node creation forms. For a single-language site it does the job.
I run a site in Danish and Swedish. On that site, almost none of those rules do anything.
My robots.txt blocks nothing. Every page carries a meta robots tag permitting indexing. Google crawls the site happily and indexes almost everything within a few days of publication.
And when an AI assistant tried to read one of my articles, it was refused.
Here is a robots.txt that appears in some form on a very large number of sites:
User-agent: *
Disallow: /admin/
Disallow: /private/
User-agent: Googlebot
Disallow: /nogoogle/
Read it the way anyone would: everyone is kept out of /admin/ and /private/, and Googlebot is additionally kept out of /nogoogle/.
Most structured data guides begin and end with Article, Product, and BreadcrumbList. For developer-focused sites — blogs covering tools, libraries, and technical how-tos — two schema types deserve far more attention: TechArticle and SoftwareApplication.
Interaction to Next Paint (INP) replaced First Input Delay (FID) as a Core Web Vital in March 2024. Where FID only measured the delay before the browser started handling the first interaction, INP measures the full latency of every interaction on a page — from user input to the next visual update.
Cumulative Layout Shift measures visual instability — how much the page content moves around while loading. Google uses it as a Core Web Vital, and a poor CLS score (above 0.25) directly affects rankings. The target is 0.1 or below at the 75th percentile.
Largest Contentful Paint measures how long it takes for the largest visible content element — usually a hero image or an H1 heading — to finish rendering in the viewport. Google's recommended threshold is under 2.5 seconds on the 75th percentile of real user loads.