For site operators

The MachineWitness crawler

Last updated 23 July 2026

If you found this page from your server logs: this is our crawler introducing itself. It is operated by Martin Schenk S.L. and it is deliberately small, polite, and identifiable.

What it does

Once per day, it requests a small, fixed set of public machine-readable files from a domain:

• /robots.txt — the crawler rules
• /ai.txt — AI-usage declarations
• /.well-known/tdmrep.json — text-and-data-mining reservations
• /llms.txt — guidance for language models
• the homepage response, for its headers and any usage meta tags (with a hard size limit)

It records exactly what was served, so that it can later be verified. It does not crawl your site's content, follow links into your application, submit forms, log in, or collect personal data beyond whatever those public files happen to contain.

How it behaves

• It identifies itself with the user-agent above and a contact address.
• It respects your robots.txt for every file except robots.txt itself — fetching the rules file is the standard convention that makes the rules checkable.
• It applies hard size and rate limits: a handful of tiny requests, once a day. The load on your server is negligible.
• It stops crawling any domain on request.

Ask to be excluded, or to have a record removed

Write to bot@machinewitness.eu. We will:

• Exclude your domain from all future crawling, on request; and / or
• Remove an existing record — the stored content is deleted and replaced by a marker (a "tombstone"), while the cryptographic proof structure is kept so the integrity of the surrounding archive still holds. See our Privacy Policy.

You can also disallow the files above in your robots.txt, which we will honour going forward.

Why we do this

These machine-readable declarations can change or disappear without a trace. A neutral, sealed daily record of what was served lets everyone — including you — later prove what your site actually declared on a given day. We are a witness, not a detective: we record and seal, and take no side. See what we do and do not do.