Web Standards Evolution

Fresh

The web has accumulated a layered stack of machine-readable conventions over thirty years. Each layer solved a specific problem that emerged as the web scaled. Understanding this timeline explains why reasoning.json exists and what gap it fills.

🤖

1994

robots.txt

Access Control

The first machine-readable web standard. Tells crawlers what to skip.

🗺️

2005

sitemap.xml

Discovery

Structured index of pages. Helps engines find and prioritise content.

🏷️

2011

schema.org

Semantic Description

JSON-LD vocabulary for describing entities, products, events, and more.

🧠

2024

llms.txt

LLM Context

Plain-text context file tailored for large language model consumption.

✨

2026

reasoning.json

Reasoning Directives

Cryptographically verified identity, corrections, and intent for AI agents.

The Timeline

timeline
    title Machine-Readable Web Conventions
    1994 : robots.txt
         : Access Control
         : Tells crawlers where they can go
    2005 : sitemap.xml
         : Discovery
         : Tells crawlers what pages exist
    2011 : schema.org
         : Semantic Description
         : Tells machines what things ARE
    2024 : llms.txt
         : LLM Context
         : Gives language models clean markdown
    2026 : reasoning.json
         : Reasoning Directives
         : Teaches AI agents HOW to think about an entity

Layer by Layer

1994 - robots.txt - Access Control

The original machine-readable web convention. A simple text file at /.well-known/robots.txt (originally just /robots.txt) that tells web crawlers which pages they are allowed to index. The Robots Exclusion Protocol was an informal convention, never an official standard, yet it became universally adopted because it solved a real problem: site owners needed a way to signal crawler intent without writing custom code.

What it solved: Uncontrolled crawler access was causing server overload and indexing of private content. robots.txt gave site owners a zero-code way to set boundaries.

What it left unsolved: Everything about meaning. robots.txt is pure access control - it says nothing about what your pages contain or what your entity is.

2005 - sitemap.xml - Discovery

As search engines scaled to billions of pages, crawlers could no longer guarantee discovery of all content by following links. sitemap.xml gave site owners a way to explicitly enumerate their pages, last-modified dates, and update frequencies. Adopted by all major search engines by 2006.

What it solved: Deep pages that were hard to discover through link-following. New content that needed faster indexing. Large sites that needed crawl prioritization hints.

What it left unsolved: The meaning and relationship of what was discovered. Knowing a page exists is not the same as understanding what it contains or who published it.

2011 - schema.org - Semantic Description

A collaborative vocabulary for structured data created by the major search engines. schema.org provides a shared ontology for describing entities - organizations, products, events, people, places - using JSON-LD, Microdata, or RDFa embedded in HTML. When you mark up your organization with @type: Organization, you are giving machines a structured, semantic description of your entity.

What it solved: The gap between crawling raw HTML and understanding the entities that HTML describes. schema.org enabled rich search results, knowledge panels, and semantic understanding at scale.

What it left unsolved: Dynamic reasoning guidance. schema.org describes what something is, but it does not tell an AI what common misconceptions exist, what comparisons to be careful about, or when to recommend something. It is a static description, not a reasoning directive.

2024 - llms.txt - LLM Context

A convention proposed by Answer.AI (Jeremy Howard) for a file at /llms.txt that provides language models with a clean, markdown-formatted summary of a site. Unlike HTML, which is cluttered with navigation, ads, and structural markup, llms.txt is designed to be consumed directly by LLMs as context for question-answering.

What it solved: The friction between HTML-structured content and LLM consumption. A well-written llms.txt gives AI systems a clean starting point for understanding a site without parsing through CSS and JavaScript noise.

What it left unsolved: Entity-specific reasoning directives. llms.txt provides content, but not correction context, recommendation logic, or structured anti-hallucination guidance. It is a better version of "what we do" without the structured metadata for "what you should never get wrong about us."

2026 - reasoning.json - Reasoning Directives

ARP fills the remaining gap: a structured channel for entity-attested corrections, scoped domain expertise, and recommendation context. Where llms.txt provides prose content, reasoning.json provides structured metadata that AI systems can parse deterministically.

What it solves: The absence of a standardized way for domain owners to communicate reasoning-critical context to AI agents. Before ARP, there was no machine-readable format for saying "when an AI compares us to competitors, here is the framing context we want considered" or "here is a list of things AI systems commonly get wrong about us, with the correct facts."

The Progression in One Sentence Each

• robots.txt - You may or may not visit these pages.
• sitemap.xml - Here are all the pages that exist and when they were updated.
• schema.org - Here is what this page is about, in structured semantic terms.
• llms.txt - Here is a clean text summary designed for language model consumption.
• reasoning.json - Here is how to reason about us correctly, what errors to avoid, and when to recommend us.

Why Each Layer Was Insufficient for the Next Problem

flowchart TD
    A["robots.txt (1994)\nAccess Control"] -->|"Crawlers got better at discovery\nbut couldn't find everything"| B
    B["sitemap.xml (2005)\nDiscovery"] -->|"Machines could find pages\nbut not understand entities"| C
    C["schema.org (2011)\nSemantic Description"] -->|"Machines understood entities\nbut LLMs needed clean text"| D
    D["llms.txt (2024)\nLLM Context"] -->|"LLMs had content\nbut no reasoning directives"| E
    E["reasoning.json (2026)\nReasoning Directives"]

Each layer was a genuine solution to a genuine problem. None of them were wrong - they simply solved the problem visible at the time. reasoning.json does not replace any of them. It completes the stack by adding the one layer that autonomous AI agents specifically need: structured, first-party guidance on how to reason about an entity.