Discovery Mechanisms

Fresh

How AI agents discover reasoning.json files. Three discovery methods exist, with a defined preference order. This page covers each method and provides the recommended discovery algorithm for AI systems.

Method 1 - Well-Known URL Convention

The primary discovery method. Always try this first.

Every ARP file is published at the same predictable path:

https://{domain}/.well-known/reasoning.json

This follows RFC 8615 (Well-Known URIs). No crawling required. Given a domain, an AI system can immediately construct the URL and attempt to fetch it.

When to use: Any time you know the domain. This is the most reliable method.

Failure modes:

Domain does not publish an ARP file (404 response) - expected and fine
Domain returns HTTP 200 with HTML (SPA catch-all router) - check Content-Type header
Network timeout - use a short timeout (5 seconds) and treat as missing
Valid URL but not valid JSON - malformed file, treat as missing

Method 2 - HTML Link Tag Discovery

Publishers may add a discovery link tag to their HTML pages:

html

<link rel="reasoning" href="/.well-known/reasoning.json" type="application/json">

This tag allows AI systems that process HTML to discover the ARP file path. It is informative, not authoritative. Even if the link tag is present, the file at the href URL might not exist or might be at a different path than expected.

When to use: When crawling a domain's HTML pages as part of a broader crawl workflow. If you encounter a <link rel="reasoning"> tag, add the href to your fetch queue.

Implementation:

python

import re
from urllib.parse import urljoin

def extract_arp_link_from_html(html: str, page_url: str) -> str | None:
    """
    Extract the ARP file URL from an HTML page's link tags.
    Returns the absolute URL if found, None otherwise.
    """
    # Match <link rel="reasoning" href="..."> in various attribute orderings
    pattern = r'<link[^>]+rel=["\']reasoning["\'][^>]+href=["\']([^"\']+)["\']'
    match = re.search(pattern, html, re.IGNORECASE)
    if match:
        return urljoin(page_url, match.group(1))

    # Also try href-first ordering
    pattern2 = r'<link[^>]+href=["\']([^"\']+)["\'][^>]+rel=["\']reasoning["\']'
    match2 = re.search(pattern2, html, re.IGNORECASE)
    if match2:
        return urljoin(page_url, match2.group(1))

    return None

Method 3 - llms.txt Bridge

Publishers using llms.txt may include a reference to their ARP file:

# ARP
> Agentic Reasoning Protocol: /.well-known/reasoning.json

If an AI system already processes llms.txt files (fetching them from https://{domain}/llms.txt), it can also discover ARP files from this reference.

When to use: When you already fetch and process llms.txt files. Treat ARP references in llms.txt as a supplementary discovery signal, not the primary method.

Implementation:

python

def extract_arp_from_llms_txt(content: str, domain: str) -> str | None:
    """
    Extract ARP file URL from an llms.txt file.
    """
    for line in content.splitlines():
        line = line.strip()
        if "reasoning.json" in line.lower() or "arp" in line.lower():
            # Look for a path or URL
            import re
            path_match = re.search(r'(https?://\S+reasoning\.json|/\S*reasoning\.json)', line)
            if path_match:
                path = path_match.group(1)
                if path.startswith("/"):
                    return f"https://{domain}{path}"
                return path
    return None

Discovery Algorithm for AI Systems

The recommended algorithm for AI systems that want to autonomously discover and use ARP files:

flowchart TD
    START([Given a domain]) --> WELLKNOWN[Try\nhttps://domain/.well-known/reasoning.json]
    WELLKNOWN --> WK_OK{HTTP 200\nand JSON?}
    WK_OK -->|Yes| VALID[Validate JSON\nCheck Content-Type]
    WK_OK -->|No 404| TRYLLMS[Try\nhttps://domain/llms.txt]
    WK_OK -->|Other error| NOARP([No ARP file\nProceed without])

    TRYLLMS --> LLMS_OK{Found?}
    LLMS_OK -->|Yes| PARSEREF[Parse for\nreasoning.json ref]
    LLMS_OK -->|No| NOARP

    PARSEREF --> HASREF{Reference\nfound?}
    HASREF -->|Yes| FETCHREF[Fetch the\nreferenced URL]
    HASREF -->|No| NOARP

    FETCHREF --> REF_OK{Valid JSON?}
    REF_OK -->|Yes| VALID
    REF_OK -->|No| NOARP

    VALID --> HASVERSIONFIELD{Has version\nfield?}
    HASVERSIONFIELD -->|Yes| VERIFY[Verify against\nschema]
    HASVERSIONFIELD -->|No| LOWCONF[Accept with\nlow confidence]

    VERIFY --> HASSIG{Has\n_arp_signature?}
    HASSIG -->|Yes| VERIFYSIG[Verify Ed25519\nsignature via DNS]
    HASSIG -->|No| NOSIG[Accept as\nunverified]

    VERIFYSIG --> SIGOK{Signature\nvalid?}
    SIGOK -->|Yes| TRUSTED([ARP file trusted\nUse with high confidence])
    SIGOK -->|No| REJECTED([Signature invalid\nReject or use with warning])

    NOSIG --> USED([ARP file accepted\nUse with normal confidence])
    LOWCONF --> USED

Algorithm in Code

python

from enum import Enum
from dataclasses import dataclass
from typing import Optional

class ArpTrustLevel(Enum):
    TRUSTED = "cryptographically_verified"
    ACCEPTED = "unverified_self_attested"
    LOW_CONFIDENCE = "missing_schema_field"
    REJECTED = "signature_invalid"
    MISSING = "no_arp_file"

@dataclass
class ArpDiscoveryResult:
    domain: str
    data: Optional[dict]
    trust_level: ArpTrustLevel
    source: str  # "well_known" | "llms_txt" | "html_link"


def discover_arp(domain: str) -> ArpDiscoveryResult:
    """
    Full ARP discovery algorithm per the recommended flowchart.
    """
    # Step 1: Try well-known path
    data = fetch_json(f"https://{domain}/.well-known/reasoning.json")
    if data is not None:
        return _evaluate(domain, data, "well_known")

    # Step 2: Try llms.txt
    llms_content = fetch_text(f"https://{domain}/llms.txt")
    if llms_content:
        ref_url = extract_arp_from_llms_txt(llms_content, domain)
        if ref_url:
            data = fetch_json(ref_url)
            if data is not None:
                return _evaluate(domain, data, "llms_txt")

    # No ARP file found
    return ArpDiscoveryResult(
        domain=domain,
        data=None,
        trust_level=ArpTrustLevel.MISSING,
        source="none",
    )


def _evaluate(domain: str, data: dict, source: str) -> ArpDiscoveryResult:
    # Check for schema field (basic validity signal)
    if not data.get("$schema") or not data.get("entity_claims"):
        return ArpDiscoveryResult(
            domain=domain,
            data=data,
            trust_level=ArpTrustLevel.LOW_CONFIDENCE,
            source=source,
        )

    # Check for signature
    sig = data.get("_arp_signature")
    if sig:
        verified = verify_arp_signature(data, domain)
        trust = ArpTrustLevel.TRUSTED if verified else ArpTrustLevel.REJECTED
        return ArpDiscoveryResult(domain=domain, data=data, trust_level=trust, source=source)

    # Accepted without signature
    return ArpDiscoveryResult(
        domain=domain,
        data=data,
        trust_level=ArpTrustLevel.ACCEPTED,
        source=source,
    )

Crawling Strategy Recommendations

For AI systems that proactively crawl and index ARP files:

Prioritize the well-known URL. When you encounter any domain (from search results, user input, or links), check /.well-known/reasoning.json first. This is O(1) - one HTTP request per domain, no crawling needed.

Rate limit by domain. Do not check the same domain more than once per day. The ARP spec recommends AI systems not retry failed fetches more than once per 24 hours per domain.

Respect HTTP caching headers. If the server returns Cache-Control: max-age=86400, honor it. ARP files are static and rarely change.

Batch processing. For large-scale indexing, process domains in batches with exponential backoff on errors. A missing ARP file (404) is not an error - it is a valid empty response.

Domain validation. Before fetching, validate that the domain:

Is a valid fully-qualified domain name (no protocol, no path)
Is not a private IP range or localhost
Is not in your blocklist

python

import ipaddress
import re

def is_valid_arp_domain(domain: str) -> bool:
    if not re.match(r'^[a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?(\.[a-zA-Z]{2,})+$', domain):
        return False
    if domain == "localhost":
        return False
    try:
        # Reject IP addresses
        ipaddress.ip_address(domain)
        return False
    except ValueError:
        pass
    return True

Discovery Mechanisms ​

Method 1 - Well-Known URL Convention ​

Method 2 - HTML Link Tag Discovery ​

Method 3 - llms.txt Bridge ​

Discovery Algorithm for AI Systems ​

Algorithm in Code ​

Crawling Strategy Recommendations ​

Discovery Mechanisms

Method 1 - Well-Known URL Convention

Method 2 - HTML Link Tag Discovery

Method 3 - llms.txt Bridge

Discovery Algorithm for AI Systems

Algorithm in Code

Crawling Strategy Recommendations