Web crawler
Token · WordPrestoBot Respects robots.txt Read-only

WordPrestoBot, our web crawler

If you found this page from a line in your server logs, hello, and thanks for checking. WordPrestoBot is the web crawler operated by Word Presto, an agentic content and SEO workflow tool. It reads public HTML to extract SEO and content signals on behalf of a Word Presto user, and nothing more.

How to control it How to recognise it
Why it visited your site

It only fetches pages on behalf of a Word Presto user.

In almost every case that is one of two things.

01

A site owner analysing their own site

Running an SEO or content review of pages they control.

02

A read-only competitive or research check

A user looking at publicly available pages to understand how a topic is covered.

It reads public HTML to extract SEO and content signals: titles, headings, metadata, links and structured data. That is it.

Behaviour

What it does, and what it does not do.

WORDPRESTOBOT DOES
request publicly accessible pages over normal HTTPS,
identify itself honestly in the User-Agent on every request,
respect robots.txt,
crawl gently, with modest request rates, timeouts and response-size limits.
WORDPRESTOBOT DOES NOT
log in, submit forms, or interact with your site,
make any changes, post content, or place orders,
attempt to access private, gated, or admin areas,
collect personal data or bypass access controls.
How to control it

One robots.txt rule controls every crawl.

All of our crawlers share the single token WordPrestoBot, so one rule is enough.

Block it entirely robots.txt 
User-agent: WordPrestoBot
Disallow: /
Block only specific areas robots.txt 
User-agent: WordPrestoBot
Disallow: /private/
Disallow: /checkout/
Or set a crawl delay robots.txt 
User-agent: WordPrestoBot
Crawl-delay: 10

Changes take effect on our next visit. We honour robots.txt, so a disallow is respected without any need to contact us.

How to recognise it

Honest on every request.

  • User-Agent token WordPrestoBot
  • Every request links back here +https://wordpresto.com/prestobot
  • Some fetches add a short note describing the specific action, for example read-only single page or sitemap and llms.txt only. Same token, same rules.
SPOOFING

If a request claims to be WordPrestoBot but does not link to this page, treat it as spoofed and feel free to block it.

not us →
Questions or concerns?

If it is causing problems, the fastest fix is the robots.txt rule above.

You can also reach us at [email protected]. Tell us the domain and we will help. If you would like WordPrestoBot to stop visiting, a disallow rule stops it on our next visit.

Curious what it was looking at?

The same content review it runs for our users, it can run for your site.

Word Presto helps writers, editors and publishing teams plan, brief, write, review and prepare content for search, readers and CMS handoff.

See what Word Presto finds on your site
PAPER · INK · SIGNAL · CONTROL